Nov 02 01:14:12.956853 ip-10-0-140-23 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 01:14:13.372139 ip-10-0-140-23 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:14:13.372139 ip-10-0-140-23 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 01:14:13.372139 ip-10-0-140-23 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:14:13.372139 ip-10-0-140-23 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:14:13.375491 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375329    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:13.375491 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375491    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:13.375491 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375494    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375498    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375500    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375503    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375506    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375508    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375511    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375514    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375516    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375520    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375523    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375526    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375529    2569 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375532    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375534    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375543    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375546    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375553    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375556    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:13.375580 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375559    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375562    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375564    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375567    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375570    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375573    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375575    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375578    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375580    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375583    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375585    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375588    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375592    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375595    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375598    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375600    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375602    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375605    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375613    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:13.376083 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375616    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375618    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375621    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375623    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375626    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375628    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375631    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375633    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375635    2569 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375638    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375641    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375643    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375646    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375649    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375651    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375654    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375656    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375659    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375661    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375664    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:13.376540 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375666    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375669    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375671    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375674    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375676    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375679    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375681    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375684    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375686    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375688    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375691    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375694    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375702    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375704    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375707    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375710    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375712    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375715    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375717    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375719    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:13.377009 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375724    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375728    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375731    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375734    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375737    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375740    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375742    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375745    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375747    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375750    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375752    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375755    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375758    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375760    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375762    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375765    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375767    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375769    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375772    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375774    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:13.377550 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375777    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375779    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375781    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375786    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375789    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.375798    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376645    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376653    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376657    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376660    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376662    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376665    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376668    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376671    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376674    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376676    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376679    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376682    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376685    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:13.378027 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376688    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376690    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376693    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376695    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376698    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376701    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376703    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376706    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376708    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376711    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376714    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376717    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376719    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376722    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376724    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376727    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376729    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376732    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376734    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:13.378520 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376737    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376740    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376742    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376745    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376747    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376750    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376752    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376754    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376757    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376760    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376763    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376765    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376769    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376771    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376774    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376776    2569 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376779    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376781    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376783    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376786    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:13.378971 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376788    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376791    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376793    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376796    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376798    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376800    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376803    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376805    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376808    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376810    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376813    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376815    2569 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376817    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376820    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376823    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376825    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376829    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376834    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376837    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376841    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:13.379461 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376844    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376847    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376850    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376854    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376856    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376859    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376861    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376863    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376867    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376871    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376873    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376876    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376879    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376881    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376884    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376886    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376889    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376891    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376894    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376896    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:13.379947 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376899    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376901    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376903    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376906    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376908    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376910    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376913    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376921    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376924    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376926    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376929    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376931    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376934    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.376937    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377459    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377466    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377470    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377474    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377478    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:13.380432 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377481    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377484    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377486    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377489    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377491    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377494    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377497    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377500    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377502    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377505    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377507    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377510    2569 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377512    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377515    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377517    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377520    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377522    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377525    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377527    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377530    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:13.380888 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377533    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377536    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377538    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377541    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377545    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377550    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377553    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377557    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377560    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377562    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377565    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377568    2569 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377571    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377573    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377575    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377578    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377580    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377583    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377585    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:13.381407 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377588    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377591    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377593    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377596    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377598    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377601    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377603    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377606    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377608    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377610    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377613    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377615    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377618    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377621    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377624    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377626    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377629    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377631    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377634    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:13.381875 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377637    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377640    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377642    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377645    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377647    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377650    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377652    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377655    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377657    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377659    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377662    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377664    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377667    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377669    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377672    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377674    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377677    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377679    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377681    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377684    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:13.382373 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377687    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377689    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377692    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377694    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377697    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377701    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377704    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377710    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377712    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377715    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377717    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377720    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377722    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377726    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377728    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377731    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377733    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377735    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377738    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:13.382847 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377740    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377743    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377746    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.377748    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377842    2569 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377849    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377855    2569 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377859    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377864    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377867    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377871    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377876    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377879    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377882    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377885    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377889    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377892    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377895    2569 flags.go:64] FLAG: --cgroup-root=""
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377899    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377902    2569 flags.go:64] FLAG: --client-ca-file=""
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377905    2569 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377908    2569 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377913    2569 flags.go:64] FLAG: --cluster-domain=""
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377916    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 01:14:13.383310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377919    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377922    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377926    2569 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377929    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377932    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377935    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377938    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377941    2569 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377944    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377947    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377951    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377953    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377957    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377960    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377963    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377966    2569 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377969    2569 flags.go:64] FLAG: --enable-server="true"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377972    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377976    2569 flags.go:64] FLAG: --event-burst="100"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377979    2569 flags.go:64] FLAG: --event-qps="50"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377982    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377985    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377988    2569 flags.go:64] FLAG: --eviction-hard=""
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377992    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 01:14:13.383863 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377995    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.377998    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378002    2569 flags.go:64] FLAG: --eviction-soft=""
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378005    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378008    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378011    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378014    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378017    2569 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378020    2569 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378023    2569 flags.go:64] FLAG: --feature-gates=""
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378027    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378031    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378034    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378037    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378040    2569 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378043    2569 flags.go:64] FLAG: --help="false"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378046    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378049    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378052    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378055    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378058    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378061    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378064    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 01:14:13.384466 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378067    2569 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378069    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378072    2569 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378075    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378078    2569 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378082    2569 flags.go:64] FLAG: --kube-reserved=""
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378084    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378087    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378090    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378093    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378096    2569 flags.go:64] FLAG: --lock-file=""
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378099    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378102    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378105    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378110    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378113    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378116    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378119    2569 flags.go:64] FLAG: --logging-format="text"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378122    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378125    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378128    2569 flags.go:64] FLAG: --manifest-url=""
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378130    2569 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378134    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378137    2569 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378141    2569 flags.go:64] FLAG: --max-pods="110"
Nov 02 01:14:13.385021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378144    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378147    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378150    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378152    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378156    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378158    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378161    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378168    2569 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378171    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378174    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378177    2569 flags.go:64] FLAG: --pod-cidr=""
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378180    2569 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378183    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378186    2569 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378189    2569 flags.go:64] FLAG: --port="10250"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378191    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378195    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-058a6e3a4adaf3bbf"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378198    2569 flags.go:64] FLAG: --qos-reserved=""
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378201    2569 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378204    2569 flags.go:64] FLAG: --register-node="true"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378207    2569 flags.go:64] FLAG: --register-with-taints=""
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378210    2569 flags.go:64] FLAG: --registry-burst="10"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378214    2569 flags.go:64] FLAG: --registry-qps="5"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378217    2569 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378220    2569 flags.go:64] FLAG: --reserved-memory=""
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378224    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 01:14:13.385728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378227    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378230    2569 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378233    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378236    2569 flags.go:64] FLAG: --runonce="false"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378239    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378241    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378244    2569 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378247    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378250    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378253    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378256    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378259    2569 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378262    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378264    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378269    2569 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378272    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378274    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378277    2569 flags.go:64] FLAG: --system-cgroups=""
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378280    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378285    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378301    2569 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378304    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378309    2569 flags.go:64] FLAG: --tls-min-version=""
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378311    2569 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378314    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 01:14:13.386676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378317    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378320    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378323    2569 flags.go:64] FLAG: --v="2"
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378327    2569 flags.go:64] FLAG: --version="false"
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378331    2569 flags.go:64] FLAG: --vmodule=""
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378335    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378338    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378464    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378468    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378471    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378473    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378476    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378479    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378481    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378484    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378486    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378489    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378491    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378494    2569 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378497    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378499    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:13.387262 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378503    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378506    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378508    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378511    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378513    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378516    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378519    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378522    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378524    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378527    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378530    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378533    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378535    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378538    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378540    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378542    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378545    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378547    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378550    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378553    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:13.387788 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378555    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378558    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378561    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378563    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378566    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378568    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378570    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378573    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378575    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378578    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378580    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378583    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378587    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378589    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378592    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378595    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378597    2569 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378601    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378604    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378606    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:13.388263 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378609    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378613    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378617    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378620    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378623    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378626    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378628    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378631    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378633    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378636    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378638    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378642    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378644    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378647    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378649    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378652    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378654    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378657    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378659    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:13.388755 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378662    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378664    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378667    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378669    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378672    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378676    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378678    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378681    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378684    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378686    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378690    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378693    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378696    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378698    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378700    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378703    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378706    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378708    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378712    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:13.389204 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378715    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378718    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378721    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378724    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378726    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378729    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378732    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378735    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378737    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378740    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378743    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378745    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378762    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378766    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.378791    2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378826    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:13.389730 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378829    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378833    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378838    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378840    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378843    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378845    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378848    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378851    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378854    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378857    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378859    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378862    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378864    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378867    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378869    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378872    2569 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378874    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378877    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378879    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:13.390106 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378882    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378884    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378887    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378889    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378892    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378895    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378898    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378900    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378902    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378905    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378908    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378910    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378912    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378915    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378917    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378921    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378924    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378926    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378929    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378931    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:13.390554 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378934    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378938    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378941    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378943    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378946    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378948    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378951    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378953    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378955    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378958    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378960    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378963    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378965    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378968    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378971    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378973    2569 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378976    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378979    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378982    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:13.391024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378985    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378988    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378992    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378995    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378997    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.378999    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379002    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379004    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379008    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379010    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379012    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379015    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379018    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379020    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379024    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379027    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379030    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379032    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379035    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:13.391527 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379037    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379040    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379042    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379045    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379047    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379050    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379052    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379055    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379057    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379060    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379062    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379065    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379068    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379070    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379073    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379075    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379077    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379080    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379084    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:13.391978 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379087    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379090    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379093    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379096    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379098    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379101    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379104    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379107    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.379110    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.386913    2569 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.386929    2569 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.386946    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.386951    2569 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387013    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387018    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387021    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387025    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387028    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387031    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387033    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:13.392457 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387036    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387039    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387042    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387044    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387047    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387050    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387052    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387055    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387058    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387060    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387063    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387065    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387068    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387071    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387073    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387076    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387079    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387086    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387089    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387091    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:13.392920 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387094    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387096    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387100    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387103    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387106    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387109    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387112    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387115    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387118    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387120    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387123    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387127    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387131    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387134    2569 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387137    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387139    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387143    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387147    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387149    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:13.393547 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387158    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387161    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387164    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387167    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387169    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387172    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387175    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387177    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387180    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387182    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387185    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387187    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387190    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387193    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387195    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387198    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387201    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387203    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387206    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387209    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:13.393995 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387212    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387214    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387217    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387219    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387221    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387224    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387226    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387229    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387231    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387233    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387236    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387239    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387241    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387250    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387252    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387255    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387257    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387260    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387262    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387265    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:13.394468 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387267    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387270    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387273    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387276    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387278    2569 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387281    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387283    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387301    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387304    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387307    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387310    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387312    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387315    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387317    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387320    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387322    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387325    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387327    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387330    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:13.394934 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387332    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.387361    2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387523    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387529    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387532    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387535    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387538    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387546    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387549    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387552    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387554    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387557    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387559    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387562    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387564    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:13.395410 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387567    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387569    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387572    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387575    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387578    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387580    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387583    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387585    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387588    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387590    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387593    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387595    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387598    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387600    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387603    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387605    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387607    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387610    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387612    2569 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387615    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:13.395780 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387618    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387620    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387623    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387625    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387627    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387635    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387638    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387640    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387643    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387645    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387648    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387650    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387653    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387655    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387658    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387660    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387663    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387666    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387668    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:13.396253 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387670    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387673    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387675    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387678    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387680    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387683    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387686    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387688    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387691    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387693    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387696    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387699    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387701    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387703    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387706    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387709    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387711    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387713    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387716    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387724    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:13.396785 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387726    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387729    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387732    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387734    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387737    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387739    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387742    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387744    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387747    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387749    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387752    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387754    2569 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387757    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387760    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387763    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387765    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387768    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387770    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387773    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387775    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:13.397279 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387778    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387781    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387786    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387789    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387793    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387797    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387800    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387803    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387806    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387808    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387811    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387813    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387821    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387824    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.387846    2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:14:13.397830 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387881    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387885    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387889    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387892    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387895    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387898    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387900    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387903    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387905    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387908    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387910    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387913    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387915    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387918    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387920    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387923    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387925    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387928    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387930    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387933    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:13.398229 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387937    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387940    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387943    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387945    2569 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387948    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387951    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387953    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387956    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387958    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387961    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387969    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387971    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387974    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387977    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387981    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387983    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387986    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387989    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387991    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387994    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:13.398821 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387996    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.387998    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388001    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388003    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388006    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388008    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388011    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388015    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388017    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388020    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388022    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388025    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388028    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388031    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388033    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388037    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388039    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388041    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388044    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:13.399321 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388046    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388049    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388052    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388054    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388063    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388066    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388069    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388071    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388074    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388076    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388079    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388081    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388084    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388086    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388089    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388091    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388093    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388096    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388098    2569 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388101    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:13.399814 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388103    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388106    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388108    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388111    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388113    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388120    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388123    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388125    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388128    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388130    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388133    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388135    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388138    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388141    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388143    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388145    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388148    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388156    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388159    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:13.400371 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388161    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388164    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388166    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388169    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388171    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388175    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388177    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:13.388180    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.388239    2569 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.392312    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.393375    2569 server.go:1014] "Starting client certificate rotation"
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.393473    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.393502    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.395433    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:14:13.400897 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.396645    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:14:13.406744 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.406729    2569 log.go:25] "Validated CRI v1 runtime API"
Nov 02 01:14:13.409526 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.409512    2569 log.go:25] "Validated CRI v1 image API"
Nov 02 01:14:13.410373 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.410343    2569 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 01:14:13.416338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.416318    2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 dab46697-ff91-48bc-a185-0a71b8409249:/dev/nvme0n1p4 f572e20b-9980-404d-947a-7909eed376a8:/dev/nvme0n1p3]
Nov 02 01:14:13.416435 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.416336    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 01:14:13.423300 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.423174    2569 manager.go:217] Machine: {Timestamp:2026-11-02 01:14:13.421922544 +0000 UTC m=+0.343047859 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099784 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d33b6981b4ee9863929337168dad1 SystemUUID:ec2d33b6-981b-4ee9-8639-29337168dad1 BootID:9870b5d9-515c-4607-aeda-a15da097825e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:aa:91:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:aa:91:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:b9:c2:dc:71:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 01:14:13.423300 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.423279    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 01:14:13.423427 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.423372    2569 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 01:14:13.424285 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.424245    2569 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 01:14:13.424444 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.424305    2569 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-23.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 01:14:13.424494 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.424448    2569 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 01:14:13.424494 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.424456    2569 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 01:14:13.424494 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.424492    2569 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:14:13.424569 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.424511    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:14:13.425178 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.425168    2569 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 01:14:13.428657 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.428644    2569 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 01:14:13.428805 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.428795    2569 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 01:14:13.428838 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.428829    2569 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 01:14:13.428865 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.428838    2569 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 01:14:13.428865 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.428849    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 01:14:13.428865 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.428857    2569 kubelet.go:405] "Adding apiserver pod source"
Nov 02 01:14:13.428954 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.428865    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 01:14:13.429169 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.429156    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:14:13.430710 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.430693    2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 01:14:13.431323 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.431305    2569 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 01:14:13.435724 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.435711    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:14:13.436452 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.436437    2569 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 01:14:13.436486 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.436470    2569 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 01:14:13.436624 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.436611    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 01:14:13.436698 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.436628    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 01:14:13.436698 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.436633    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 01:14:13.436698 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.436639    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 01:14:13.436698 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.436645    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 01:14:13.436698 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.436650    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 01:14:13.437068 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.437060    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 01:14:13.437101 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.437069    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 01:14:13.437101 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.437076    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 01:14:13.437101 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.437082    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 01:14:13.437101 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.437090    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 01:14:13.437267 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.437150    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 01:14:13.437802 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.437792    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 01:14:13.437802 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.437802    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 01:14:13.441604 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.441591    2569 server.go:1263] "Started kubelet"
Nov 02 01:14:13.441708 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.441672    2569 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 01:14:13.441882 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.441778    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 01:14:13.441991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.441911    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 01:14:13.442632 ip-10-0-140-23 systemd[1]: Started Kubernetes Kubelet.
Nov 02 01:14:13.443219 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.443155    2569 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 01:14:13.443681 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.443669    2569 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 01:14:13.446410 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.446353    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-23.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 01:14:13.448105 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.448088    2569 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 01:14:13.450580 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.450563    2569 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 01:14:13.450580 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.450579    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 01:14:13.450708 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.450654    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:13.451444 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.451235    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 01:14:13.452057 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.452023    2569 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 01:14:13.452057 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.452054    2569 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 01:14:13.452215 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.452204    2569 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 01:14:13.452215 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.452215    2569 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 01:14:13.453753 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.453618    2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 01:14:13.453753 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.453686    2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-23.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 01:14:13.454463 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.454448    2569 factory.go:55] Registering systemd factory
Nov 02 01:14:13.454545 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.454465    2569 factory.go:223] Registration of the systemd container factory successfully
Nov 02 01:14:13.454706 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.454688    2569 factory.go:153] Registering CRI-O factory
Nov 02 01:14:13.454706 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.454706    2569 factory.go:223] Registration of the crio container factory successfully
Nov 02 01:14:13.454838 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.454758    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 01:14:13.454838 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.454783    2569 factory.go:103] Registering Raw factory
Nov 02 01:14:13.454838 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.454797    2569 manager.go:1199] Started watching for new ooms in manager
Nov 02 01:14:13.455421 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.453591    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-23.us-east-2.compute.internal.18e4155ff2e6729b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-23.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-23.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 01:14:13.441565339 +0000 UTC m=+0.362690654,LastTimestamp:2026-11-02 01:14:13.441565339 +0000 UTC m=+0.362690654,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-23.us-east-2.compute.internal,}"
Nov 02 01:14:13.455976 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.455962    2569 manager.go:319] Starting recovery of all containers
Nov 02 01:14:13.464310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.463086    2569 manager.go:324] Recovery completed
Nov 02 01:14:13.464979 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.464949    2569 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-23.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 01:14:13.465060 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.465042    2569 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 02 01:14:13.469718 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.469608    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:14:13.471689 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.471664    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:14:13.471754 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.471702    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:14:13.471754 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.471712    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:14:13.472262 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.472244    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:14:13.472354 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.472244    2569 cpu_manager.go:226] "Starting" policy="none"
Nov 02 01:14:13.472354 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.472322    2569 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 01:14:13.472354 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.472346    2569 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 01:14:13.472902 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.472887    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:14:13.472968 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.472914    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:14:13.472968 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.472927    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:14:13.474501 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.474488    2569 policy_none.go:50] "Start"
Nov 02 01:14:13.474539 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.474504    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 01:14:13.474539 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.474521    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 01:14:13.476523 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.476511    2569 policy_none.go:44] "Start"
Nov 02 01:14:13.482476 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.482455    2569 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 01:14:13.482600 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.482524    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-23.us-east-2.compute.internal.18e4155ff4b216b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-23.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-23.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-23.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 01:14:13.471688369 +0000 UTC m=+0.392813687,LastTimestamp:2026-11-02 01:14:13.471688369 +0000 UTC m=+0.392813687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-23.us-east-2.compute.internal,}"
Nov 02 01:14:13.484956 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.484899    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-23.us-east-2.compute.internal.18e4155ff4b25fce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-23.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-23.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-23.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 01:14:13.471707086 +0000 UTC m=+0.392832401,LastTimestamp:2026-11-02 01:14:13.471707086 +0000 UTC m=+0.392832401,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-23.us-east-2.compute.internal,}"
Nov 02 01:14:13.487676 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.487657    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vrzxb"
Nov 02 01:14:13.489711 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.489658    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-23.us-east-2.compute.internal.18e4155ff4b282e7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-23.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-23.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-23.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 01:14:13.471716071 +0000 UTC m=+0.392841386,LastTimestamp:2026-11-02 01:14:13.471716071 +0000 UTC m=+0.392841386,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-23.us-east-2.compute.internal,}"
Nov 02 01:14:13.498232 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.498174    2569 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-23.us-east-2.compute.internal.18e4155ff4b216b1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-23.us-east-2.compute.internal.18e4155ff4b216b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-23.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-23.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-23.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 01:14:13.471688369 +0000 UTC m=+0.392813687,LastTimestamp:2026-11-02 01:14:13.47289928 +0000 UTC m=+0.394024596,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-23.us-east-2.compute.internal,}"
Nov 02 01:14:13.499354 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.499261    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vrzxb"
Nov 02 01:14:13.502335 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.502258    2569 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-23.us-east-2.compute.internal.18e4155ff4b25fce\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-23.us-east-2.compute.internal.18e4155ff4b25fce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-23.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-23.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-23.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 01:14:13.471707086 +0000 UTC m=+0.392832401,LastTimestamp:2026-11-02 01:14:13.472920426 +0000 UTC m=+0.394045742,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-23.us-east-2.compute.internal,}"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.520894    2569 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.523316    2569 manager.go:341] "Starting Device Plugin manager"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.523340    2569 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.523349    2569 server.go:87] "Starting device plugin registration server"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.523549    2569 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.523558    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.523656    2569 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.523724    2569 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.523732    2569 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.524204    2569 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.524246    2569 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.530918    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.532217    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.532234    2569 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.532251    2569 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.532282    2569 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 01:14:13.542272 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.535792    2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:14:13.624632 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.624559    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:14:13.625565 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.625549    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:14:13.625661 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.625585    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:14:13.625661 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.625599    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:14:13.625661 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.625631    2569 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.632457 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.632427    2569 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"]
Nov 02 01:14:13.632531 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.632486    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:14:13.633281 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.633265    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:14:13.633374 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.633307    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:14:13.633374 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.633318    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:14:13.633469 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.633454    2569 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.633519 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.633482    2569 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-23.us-east-2.compute.internal\": node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:13.635434 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.635422    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:14:13.635568 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.635554    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.635622 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.635590    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:14:13.636151 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.636135    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:14:13.636211 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.636149    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:14:13.636211 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.636167    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:14:13.636211 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.636172    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:14:13.636211 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.636183    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:14:13.636211 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.636185    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:14:13.638219 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.638203    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.638322 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.638238    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:14:13.638887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.638872    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:14:13.639015 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.638900    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:14:13.639015 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.638915    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:14:13.652760 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.652736    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.652845 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.652762    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a10306333dfa6bc03e77a9c914cffe89-config\") pod \"kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"a10306333dfa6bc03e77a9c914cffe89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.652845 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.652797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.658665 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.658651    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:13.662982 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.662961    2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found" node="ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.669856 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.669840    2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found" node="ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.752948 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.752917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a10306333dfa6bc03e77a9c914cffe89-config\") pod \"kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"a10306333dfa6bc03e77a9c914cffe89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.752948 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.752951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.753159 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.752974    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.753159 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.753025    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.753159 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.753027    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a10306333dfa6bc03e77a9c914cffe89-config\") pod \"kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"a10306333dfa6bc03e77a9c914cffe89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.753159 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.753027    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.759037 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.759019    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:13.859557 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.859523    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:13.960405 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:13.960342    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:13.967500 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.967486    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:13.972715 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:13.972692    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:14.060483 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.060455    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:14.160931 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.160911    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:14.261504 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.261453    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:14.361956 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.361932    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-23.us-east-2.compute.internal\" not found"
Nov 02 01:14:14.393111 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.393089    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 01:14:14.393738 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.393214    2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:14:14.428024 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.427997    2569 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:14:14.430066 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.430054    2569 apiserver.go:52] "Watching apiserver"
Nov 02 01:14:14.438996 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.438980    2569 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 01:14:14.439258 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.439241    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-nk8xc","openshift-cluster-node-tuning-operator/tuned-r2dhn","openshift-multus/multus-fps5b","openshift-dns/node-resolver-2wxd5","openshift-image-registry/node-ca-hlw47","openshift-multus/multus-additional-cni-plugins-z9bzl","openshift-ovn-kubernetes/ovnkube-node-75mz6","kube-system/global-pull-secret-syncer-nm87s","kube-system/konnectivity-agent-dqdrz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh","openshift-network-diagnostics/network-check-target-tkzsw","openshift-multus/network-metrics-daemon-2hkrs"]
Nov 02 01:14:14.441624 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.441607    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.443599 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.443579    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.445303 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.445267    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:14.445303 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.445279    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 01:14:14.445303 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.445301    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67z2p\""
Nov 02 01:14:14.445501 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.445283    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:14.445580 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.445566    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.446444 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.446426    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:14.446554 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.446537    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:14.446616 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.446589    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlhpz\""
Nov 02 01:14:14.447640 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.447626    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.448494 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.448480    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-87cgr\""
Nov 02 01:14:14.448556 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.448539    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 01:14:14.448775 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.448765    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 01:14:14.448824 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.448813    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 01:14:14.448860 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.448825    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 01:14:14.449662 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.449649    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.450690 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.450675    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 01:14:14.450690 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.450684    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 01:14:14.450811 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.450697    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 01:14:14.450811 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.450729    2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:14.450811 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.450766    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-85fmt\""
Nov 02 01:14:14.451765 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.451751    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.452722 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.452706    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 01:14:14.452856 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.452841    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 01:14:14.452856 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.452849    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 01:14:14.452992 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.452980    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8rk5x\""
Nov 02 01:14:14.454059 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.454044    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.454666 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.454645    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 01:14:14.454830 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.454817    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 01:14:14.455053 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.455037    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5kn9m\""
Nov 02 01:14:14.456300 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456265    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-kubernetes\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.456367 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456314    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.456367 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456325    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:14.456367 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456353    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-bin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.456521 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456393    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-tmp-dir\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.456521 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.456402    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:14.456521 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456417    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.456521 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnfhv\" (UniqueName: \"kubernetes.io/projected/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-kube-api-access-dnfhv\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.456521 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-sys\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.456712 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456524    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-lib-modules\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.456712 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456568    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-socket-dir-parent\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.456712 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456596    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-multus\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.456712 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456620    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9v2\" (UniqueName: \"kubernetes.io/projected/eef42078-d140-4216-8ee9-0cc6f0809a33-kube-api-access-mm9v2\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.456712 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456637    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztlqj\" (UniqueName: \"kubernetes.io/projected/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-kube-api-access-ztlqj\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.456712 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456678    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-host-slash\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.456712 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456708    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysconfig\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.456991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456729    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-var-lib-kubelet\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.456991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-host\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.456991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-netns\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.456991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-etc-kubernetes\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.456991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwlm2\" (UniqueName: \"kubernetes.io/projected/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-kube-api-access-rwlm2\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.456991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456870    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-cnibin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.456991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456915    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-kubelet\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.456991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4afb19d-ffe5-4011-81a0-5761db34bd75-host\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.456991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.456979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4afb19d-ffe5-4011-81a0-5761db34bd75-serviceca\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.457338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cnibin\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.457338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457043    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.457338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-iptables-alerter-script\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.457338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457130    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-run\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.457338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457165    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-tmp\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.457338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.457338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-cni-binary-copy\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.457338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-k8s-cni-cncf-io\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.457338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457300    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.457338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457328    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-conf\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.457669 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-systemd\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.457669 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457406    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-modprobe-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.457669 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457451    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-hostroot\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.457669 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-conf-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.457669 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457548    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-multus-certs\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.457669 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-system-cni-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.457669 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-etc-tuned\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.457669 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457616    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4h9\" (UniqueName: \"kubernetes.io/projected/5d45205b-f684-4607-b594-0352c721a4f7-kube-api-access-4l4h9\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.457669 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-system-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.458260 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457678    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-daemon-config\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.458260 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457764    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-hosts-file\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.458260 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn89l\" (UniqueName: \"kubernetes.io/projected/c4afb19d-ffe5-4011-81a0-5761db34bd75-kube-api-access-cn89l\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.458260 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.457842    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.458677 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.458660    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:14.460976 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.460958    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.461071 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.460958    2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:14:14.461132 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.461120    2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:14.463318 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.463303    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:14.463400 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.463352    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:14.465405 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.465386    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"]
Nov 02 01:14:14.465505 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.465469    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:14.465564 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.465530    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:14.476916 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.476897    2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:14:14.477114 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.477093    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"]
Nov 02 01:14:14.505657 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.505611    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:09:14 +0000 UTC" deadline="2028-07-30 10:21:22.905657364 +0000 UTC"
Nov 02 01:14:14.505657 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.505655    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15273h7m8.400004948s"
Nov 02 01:14:14.536431 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.536406    2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:14:14.536431 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.536419    2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 01:14:14.536623 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.536484    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-npmcc"
Nov 02 01:14:14.545678 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.545660    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-npmcc"
Nov 02 01:14:14.545780 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.545764    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 01:14:14.545838 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.545822    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 01:14:14.545910 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.545889    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 01:14:14.545910 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.545901    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pl9l\""
Nov 02 01:14:14.546085 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.545884    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 01:14:14.546085 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.546034    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 01:14:14.546211 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.546091    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 01:14:14.546211 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.546112    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 01:14:14.546211 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.546116    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 01:14:14.546211 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.546199    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-75lmf\""
Nov 02 01:14:14.546385 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.546200    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 01:14:14.546385 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.546224    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tmdq2\""
Nov 02 01:14:14.546385 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.546228    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 01:14:14.546385 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.546240    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 01:14:14.551915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.551901    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 01:14:14.558022 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558000    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-kubelet\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.558132 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558035    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4afb19d-ffe5-4011-81a0-5761db34bd75-host\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.558132 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cnibin\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.558132 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558094    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:14.558132 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558096    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-kubelet\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558138    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4afb19d-ffe5-4011-81a0-5761db34bd75-host\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558172    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-var-lib-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558181    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cnibin\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558199    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-config\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558233    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5whv\" (UniqueName: \"kubernetes.io/projected/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-kube-api-access-n5whv\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558254    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-iptables-alerter-script\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558279    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-run\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558331    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558427    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-run\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558454    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.558472 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558467    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-cni-binary-copy\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-k8s-cni-cncf-io\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558543    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-systemd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558566    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-ovn\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558596    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-bin\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558626    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-k8s-cni-cncf-io\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558663    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96b08fc5-cc99-4228-9a22-1e57c7fed18b-agent-certs\") pod \"konnectivity-agent-dqdrz\" (UID: \"96b08fc5-cc99-4228-9a22-1e57c7fed18b\") " pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558687    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-node-log\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558718    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cphg\" (UniqueName: \"kubernetes.io/projected/68a83bf9-59f7-42a3-aeb0-efc8691a6644-kube-api-access-2cphg\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558739    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-socket-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558777    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-modprobe-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558806    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-hostroot\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-conf-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558870    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-conf-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558884    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-hostroot\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-multus-certs\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558911    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-iptables-alerter-script\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.559063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558947    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-cni-binary-copy\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558947    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-modprobe-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558950    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.558982    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-multus-certs\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559025    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-script-lib\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-sys-fs\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559103    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-etc-tuned\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-system-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-daemon-config\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559169    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn89l\" (UniqueName: \"kubernetes.io/projected/c4afb19d-ffe5-4011-81a0-5761db34bd75-kube-api-access-cn89l\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559192    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559198    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-system-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559216    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559241    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-bin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559302    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-bin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-etc-selinux\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559366    2569 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559385    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnfhv\" (UniqueName: \"kubernetes.io/projected/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-kube-api-access-dnfhv\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.559823 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559396    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559500    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-sys\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztlqj\" (UniqueName: \"kubernetes.io/projected/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-kube-api-access-ztlqj\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559547    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-sys\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-dbus\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559583    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-device-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559607    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-host-slash\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559620    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-daemon-config\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysconfig\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559648    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-host\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559671    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-host-slash\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559680    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-netns\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559715    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-etc-kubernetes\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559725    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysconfig\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwlm2\" (UniqueName: \"kubernetes.io/projected/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-kube-api-access-rwlm2\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559749    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-netns\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559788    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-host\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559815    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.560558 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559824    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-etc-kubernetes\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559893    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-cnibin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4afb19d-ffe5-4011-81a0-5761db34bd75-serviceca\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559932    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-kubelet\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559940    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-cnibin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559950    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-netns\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.559969    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-tmp\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560019    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560096    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-conf\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-systemd\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-system-cni-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560169    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560192    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-slash\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560207    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-log-socket\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560228    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4h9\" (UniqueName: \"kubernetes.io/projected/5d45205b-f684-4607-b594-0352c721a4f7-kube-api-access-4l4h9\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560232    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-conf\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-hosts-file\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.561402 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560323    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560353    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-systemd\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-kubelet-config\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560385    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovn-node-metrics-cert\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560394    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-hosts-file\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ssf\" (UniqueName: \"kubernetes.io/projected/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kube-api-access-t4ssf\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-kubernetes\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560532    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-tmp-dir\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560557    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96b08fc5-cc99-4228-9a22-1e57c7fed18b-konnectivity-ca\") pod \"konnectivity-agent-dqdrz\" (UID: \"96b08fc5-cc99-4228-9a22-1e57c7fed18b\") " pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560625    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-systemd-units\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560683    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560753    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-kubernetes\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-registration-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560847    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.561887 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-lib-modules\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-socket-dir-parent\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.560973    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4afb19d-ffe5-4011-81a0-5761db34bd75-serviceca\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561000    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-lib-modules\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561034    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-socket-dir-parent\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561054    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-multus\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9v2\" (UniqueName: \"kubernetes.io/projected/eef42078-d140-4216-8ee9-0cc6f0809a33-kube-api-access-mm9v2\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561106    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-multus\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-etc-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561137    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561157    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-tmp-dir\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561159    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-env-overrides\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561180    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-system-cni-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561217    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-var-lib-kubelet\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561267    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-netd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561331    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-var-lib-kubelet\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.562495 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.561475    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.563052 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.562411    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-tmp\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.563052 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.562575    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-etc-tuned\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.574024 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.573996    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwlm2\" (UniqueName: \"kubernetes.io/projected/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-kube-api-access-rwlm2\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.574368 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.574345    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4h9\" (UniqueName: \"kubernetes.io/projected/5d45205b-f684-4607-b594-0352c721a4f7-kube-api-access-4l4h9\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.574452 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.574435    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztlqj\" (UniqueName: \"kubernetes.io/projected/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-kube-api-access-ztlqj\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.574515 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.574468    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9v2\" (UniqueName: \"kubernetes.io/projected/eef42078-d140-4216-8ee9-0cc6f0809a33-kube-api-access-mm9v2\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.574786 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.574770    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnfhv\" (UniqueName: \"kubernetes.io/projected/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-kube-api-access-dnfhv\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.575397 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.575378    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn89l\" (UniqueName: \"kubernetes.io/projected/c4afb19d-ffe5-4011-81a0-5761db34bd75-kube-api-access-cn89l\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.662312 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662266    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-netd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662459 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662317    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662459 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:14.662459 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-var-lib-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662459 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662358    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-netd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662459 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-config\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662459 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5whv\" (UniqueName: \"kubernetes.io/projected/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-kube-api-access-n5whv\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:14.662459 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662437    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-systemd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662459 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-ovn\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662456    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-var-lib-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-bin\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662456    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662517    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-systemd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96b08fc5-cc99-4228-9a22-1e57c7fed18b-agent-certs\") pod \"konnectivity-agent-dqdrz\" (UID: \"96b08fc5-cc99-4228-9a22-1e57c7fed18b\") " pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662528    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-ovn\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662560    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-node-log\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662586    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-bin\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662586    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cphg\" (UniqueName: \"kubernetes.io/projected/68a83bf9-59f7-42a3-aeb0-efc8691a6644-kube-api-access-2cphg\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-socket-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-script-lib\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662627    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-node-log\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-sys-fs\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662737    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662755    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-etc-selinux\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.662921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662762    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662785    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-dbus\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662799    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-sys-fs\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-device-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662833    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-kubelet\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662869    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-dbus\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-netns\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662886    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-etc-selinux\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662913    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-kubelet\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662943    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-device-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662924    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-netns\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.662976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-slash\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-log-socket\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-kubelet-config\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663058    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovn-node-metrics-cert\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.663851 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663076    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ssf\" (UniqueName: \"kubernetes.io/projected/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kube-api-access-t4ssf\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663094    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-log-socket\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.663110    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663147    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-slash\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663171    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-kubelet-config\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663326    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96b08fc5-cc99-4228-9a22-1e57c7fed18b-konnectivity-ca\") pod \"konnectivity-agent-dqdrz\" (UID: \"96b08fc5-cc99-4228-9a22-1e57c7fed18b\") " pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-systemd-units\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663388    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-socket-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663420    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.663484    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret podName:7b6da542-d39f-4a29-9849-08164ec387e0 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:15.163444443 +0000 UTC m=+2.084569746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret") pod "global-pull-secret-syncer-nm87s" (UID: "7b6da542-d39f-4a29-9849-08164ec387e0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663556    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-config\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663616    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-systemd-units\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663670    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663723    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-script-lib\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663935    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-registration-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663971    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.663999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-etc-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.664619 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.664023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-env-overrides\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.665221 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.664032    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-registration-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.665221 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.664077    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:14.665221 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.664070    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-etc-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.665221 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.664119    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:15.164108094 +0000 UTC m=+2.085233396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:14.665221 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.664509    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96b08fc5-cc99-4228-9a22-1e57c7fed18b-konnectivity-ca\") pod \"konnectivity-agent-dqdrz\" (UID: \"96b08fc5-cc99-4228-9a22-1e57c7fed18b\") " pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:14.665221 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.664596    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-env-overrides\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.665652 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.665633    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovn-node-metrics-cert\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.665856 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.665839    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96b08fc5-cc99-4228-9a22-1e57c7fed18b-agent-certs\") pod \"konnectivity-agent-dqdrz\" (UID: \"96b08fc5-cc99-4228-9a22-1e57c7fed18b\") " pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:14.672832 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.672814    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:14:14.672832 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.672830    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:14:14.672987 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.672840    2569 projected.go:196] Error preparing data for projected volume kube-api-access-6bq9l for pod openshift-network-diagnostics/network-check-target-tkzsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:14.672987 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:14.672875    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l podName:12cc7107-489a-4616-a157-e6321337b9be nodeName:}" failed. No retries permitted until 2026-11-02 01:14:15.172864574 +0000 UTC m=+2.093989880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6bq9l" (UniqueName: "kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l") pod "network-check-target-tkzsw" (UID: "12cc7107-489a-4616-a157-e6321337b9be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:14.675218 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.675200    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5whv\" (UniqueName: \"kubernetes.io/projected/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-kube-api-access-n5whv\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:14.675396 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.675381    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ssf\" (UniqueName: \"kubernetes.io/projected/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kube-api-access-t4ssf\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.675441 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.675406    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cphg\" (UniqueName: \"kubernetes.io/projected/68a83bf9-59f7-42a3-aeb0-efc8691a6644-kube-api-access-2cphg\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.705795 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.705753    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699c49bf3206a78424a5f5d5e4f1b0bc.slice/crio-1c2fae246ea9e2354173d0be93469c0dce2ae78e79e3f6f37f8d8ff5272bbdbb WatchSource:0}: Error finding container 1c2fae246ea9e2354173d0be93469c0dce2ae78e79e3f6f37f8d8ff5272bbdbb: Status 404 returned error can't find the container with id 1c2fae246ea9e2354173d0be93469c0dce2ae78e79e3f6f37f8d8ff5272bbdbb
Nov 02 01:14:14.706193 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.706175    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10306333dfa6bc03e77a9c914cffe89.slice/crio-8a29820c4984e659393d0f60e28378c39745227abfecbb12e5df400a06af2bd3 WatchSource:0}: Error finding container 8a29820c4984e659393d0f60e28378c39745227abfecbb12e5df400a06af2bd3: Status 404 returned error can't find the container with id 8a29820c4984e659393d0f60e28378c39745227abfecbb12e5df400a06af2bd3
Nov 02 01:14:14.710698 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.710683    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:14:14.774107 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.774039    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:14.784024 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.784000    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7ea8b0_06fd_457e_804b_da8c4f73edfc.slice/crio-79b9e8663e48dfa9bec608ecd319d8aef0f9a899aa0b80e0866881a57373953b WatchSource:0}: Error finding container 79b9e8663e48dfa9bec608ecd319d8aef0f9a899aa0b80e0866881a57373953b: Status 404 returned error can't find the container with id 79b9e8663e48dfa9bec608ecd319d8aef0f9a899aa0b80e0866881a57373953b
Nov 02 01:14:14.796355 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.796336    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:14.802224 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.802204    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fps5b"
Nov 02 01:14:14.804044 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.804025    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d45205b_f684_4607_b594_0352c721a4f7.slice/crio-0b094af6c779e56c479058b2de857aef2702b203acf7e5892abd018146094c7e WatchSource:0}: Error finding container 0b094af6c779e56c479058b2de857aef2702b203acf7e5892abd018146094c7e: Status 404 returned error can't find the container with id 0b094af6c779e56c479058b2de857aef2702b203acf7e5892abd018146094c7e
Nov 02 01:14:14.810122 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.810098    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef42078_d140_4216_8ee9_0cc6f0809a33.slice/crio-10fb972b8d11e03e3c10123b38da659288991e0786c0a985e169cc2a79c65c24 WatchSource:0}: Error finding container 10fb972b8d11e03e3c10123b38da659288991e0786c0a985e169cc2a79c65c24: Status 404 returned error can't find the container with id 10fb972b8d11e03e3c10123b38da659288991e0786c0a985e169cc2a79c65c24
Nov 02 01:14:14.821686 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.821670    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:14.830935 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.830913    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c0fff4_df96_4db9_a0b1_ddcb4c8d028f.slice/crio-94b89615a53092de60a9fb64c3500c8748b614bcee93733bc4eee294846f1b4b WatchSource:0}: Error finding container 94b89615a53092de60a9fb64c3500c8748b614bcee93733bc4eee294846f1b4b: Status 404 returned error can't find the container with id 94b89615a53092de60a9fb64c3500c8748b614bcee93733bc4eee294846f1b4b
Nov 02 01:14:14.834260 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.834240    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:14.842243 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.842218    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4afb19d_ffe5_4011_81a0_5761db34bd75.slice/crio-cac6b082d49b82a25c0009cc24f65f321bf19a23b063a79ac924cc87450c3e94 WatchSource:0}: Error finding container cac6b082d49b82a25c0009cc24f65f321bf19a23b063a79ac924cc87450c3e94: Status 404 returned error can't find the container with id cac6b082d49b82a25c0009cc24f65f321bf19a23b063a79ac924cc87450c3e94
Nov 02 01:14:14.874369 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.874341    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:14.883773 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.883749    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad10cee_ccd8_424d_8e5b_8eed99b3881e.slice/crio-9da9a4bca090f45342cbeb031e9fd07da4ece5a9e0698a480b520c77cec098f3 WatchSource:0}: Error finding container 9da9a4bca090f45342cbeb031e9fd07da4ece5a9e0698a480b520c77cec098f3: Status 404 returned error can't find the container with id 9da9a4bca090f45342cbeb031e9fd07da4ece5a9e0698a480b520c77cec098f3
Nov 02 01:14:14.901230 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.901214    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:14.907212 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.907199    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:14.910453 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.910430    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b08fc5_cc99_4228_9a22_1e57c7fed18b.slice/crio-959d6c6c6831aca9731a1a9a29a3df47943b5c55e37372d60107742c14547efc WatchSource:0}: Error finding container 959d6c6c6831aca9731a1a9a29a3df47943b5c55e37372d60107742c14547efc: Status 404 returned error can't find the container with id 959d6c6c6831aca9731a1a9a29a3df47943b5c55e37372d60107742c14547efc
Nov 02 01:14:14.913349 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.913328    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:14.915779 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.915758    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a83bf9_59f7_42a3_aeb0_efc8691a6644.slice/crio-f0dcefa6154dda1cd27789613274e478e25dbe925f881ddf2788705d8ffe2b14 WatchSource:0}: Error finding container f0dcefa6154dda1cd27789613274e478e25dbe925f881ddf2788705d8ffe2b14: Status 404 returned error can't find the container with id f0dcefa6154dda1cd27789613274e478e25dbe925f881ddf2788705d8ffe2b14
Nov 02 01:14:14.921931 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:14.921907    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f29f286_e8a8_41e8_b0b7_c18dacdbdd83.slice/crio-7fb18e18bdef725db03e0cdccb78cdbb10e2a4be6a9bf50facafb13b85813dee WatchSource:0}: Error finding container 7fb18e18bdef725db03e0cdccb78cdbb10e2a4be6a9bf50facafb13b85813dee: Status 404 returned error can't find the container with id 7fb18e18bdef725db03e0cdccb78cdbb10e2a4be6a9bf50facafb13b85813dee
Nov 02 01:14:14.955170 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:14.955147    2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:14:15.169813 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.169736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:15.169813 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.169777    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:15.170038 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:15.169897    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:15.170038 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:15.169898    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:15.170038 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:15.169960    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:16.169945895 +0000 UTC m=+3.091071196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:15.170038 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:15.169974    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret podName:7b6da542-d39f-4a29-9849-08164ec387e0 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:16.16996786 +0000 UTC m=+3.091093162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret") pod "global-pull-secret-syncer-nm87s" (UID: "7b6da542-d39f-4a29-9849-08164ec387e0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:15.240524 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.240496    2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:14:15.270941 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.270910    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:15.271084 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:15.271041    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:14:15.271084 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:15.271056    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:14:15.271084 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:15.271065    2569 projected.go:196] Error preparing data for projected volume kube-api-access-6bq9l for pod openshift-network-diagnostics/network-check-target-tkzsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:15.271181 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:15.271113    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l podName:12cc7107-489a-4616-a157-e6321337b9be nodeName:}" failed. No retries permitted until 2026-11-02 01:14:16.271099738 +0000 UTC m=+3.192225053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bq9l" (UniqueName: "kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l") pod "network-check-target-tkzsw" (UID: "12cc7107-489a-4616-a157-e6321337b9be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:15.548535 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.548368    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:09:15 +0000 UTC" deadline="2028-08-01 23:01:13.974148484 +0000 UTC"
Nov 02 01:14:15.548535 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.548439    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15333h46m58.425714654s"
Nov 02 01:14:15.573692 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.573463    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6" event={"ID":"68a83bf9-59f7-42a3-aeb0-efc8691a6644","Type":"ContainerStarted","Data":"f0dcefa6154dda1cd27789613274e478e25dbe925f881ddf2788705d8ffe2b14"}
Nov 02 01:14:15.585229 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.584920    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bzl" event={"ID":"3ad10cee-ccd8-424d-8e5b-8eed99b3881e","Type":"ContainerStarted","Data":"9da9a4bca090f45342cbeb031e9fd07da4ece5a9e0698a480b520c77cec098f3"}
Nov 02 01:14:15.601582 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.601546    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlw47" event={"ID":"c4afb19d-ffe5-4011-81a0-5761db34bd75","Type":"ContainerStarted","Data":"cac6b082d49b82a25c0009cc24f65f321bf19a23b063a79ac924cc87450c3e94"}
Nov 02 01:14:15.606929 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.606897    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fps5b" event={"ID":"eef42078-d140-4216-8ee9-0cc6f0809a33","Type":"ContainerStarted","Data":"10fb972b8d11e03e3c10123b38da659288991e0786c0a985e169cc2a79c65c24"}
Nov 02 01:14:15.614523 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.614492    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r2dhn" event={"ID":"5d45205b-f684-4607-b594-0352c721a4f7","Type":"ContainerStarted","Data":"0b094af6c779e56c479058b2de857aef2702b203acf7e5892abd018146094c7e"}
Nov 02 01:14:15.620118 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.620089    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal" event={"ID":"a10306333dfa6bc03e77a9c914cffe89","Type":"ContainerStarted","Data":"8a29820c4984e659393d0f60e28378c39745227abfecbb12e5df400a06af2bd3"}
Nov 02 01:14:15.636127 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.636094    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal" event={"ID":"699c49bf3206a78424a5f5d5e4f1b0bc","Type":"ContainerStarted","Data":"1c2fae246ea9e2354173d0be93469c0dce2ae78e79e3f6f37f8d8ff5272bbdbb"}
Nov 02 01:14:15.654831 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.654793    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh" event={"ID":"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83","Type":"ContainerStarted","Data":"7fb18e18bdef725db03e0cdccb78cdbb10e2a4be6a9bf50facafb13b85813dee"}
Nov 02 01:14:15.660657 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.660615    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dqdrz" event={"ID":"96b08fc5-cc99-4228-9a22-1e57c7fed18b","Type":"ContainerStarted","Data":"959d6c6c6831aca9731a1a9a29a3df47943b5c55e37372d60107742c14547efc"}
Nov 02 01:14:15.663021 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.662989    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2wxd5" event={"ID":"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f","Type":"ContainerStarted","Data":"94b89615a53092de60a9fb64c3500c8748b614bcee93733bc4eee294846f1b4b"}
Nov 02 01:14:15.664732 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:15.664674    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nk8xc" event={"ID":"ce7ea8b0-06fd-457e-804b-da8c4f73edfc","Type":"ContainerStarted","Data":"79b9e8663e48dfa9bec608ecd319d8aef0f9a899aa0b80e0866881a57373953b"}
Nov 02 01:14:16.179512 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:16.179472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:16.179717 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:16.179529    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:16.179717 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.179699    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:16.179854 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.179768    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:18.179750238 +0000 UTC m=+5.100875559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:16.180217 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.180193    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:16.180323 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.180254    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret podName:7b6da542-d39f-4a29-9849-08164ec387e0 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:18.180237236 +0000 UTC m=+5.101362546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret") pod "global-pull-secret-syncer-nm87s" (UID: "7b6da542-d39f-4a29-9849-08164ec387e0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:16.280474 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:16.280390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:16.280655 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.280608    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:14:16.280655 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.280629    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:14:16.280655 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.280651    2569 projected.go:196] Error preparing data for projected volume kube-api-access-6bq9l for pod openshift-network-diagnostics/network-check-target-tkzsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:16.280829 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.280716    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l podName:12cc7107-489a-4616-a157-e6321337b9be nodeName:}" failed. No retries permitted until 2026-11-02 01:14:18.28069867 +0000 UTC m=+5.201823981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bq9l" (UniqueName: "kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l") pod "network-check-target-tkzsw" (UID: "12cc7107-489a-4616-a157-e6321337b9be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:16.532976 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:16.532894    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:16.533148 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.533032    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:16.533522 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:16.533499    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:16.533601 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:16.533536    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:16.533659 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.533633    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:16.533801 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:16.533729    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:16.549341 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:16.549285    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:09:15 +0000 UTC" deadline="2028-06-04 13:17:04.965382803 +0000 UTC"
Nov 02 01:14:16.549341 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:16.549339    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13932h2m48.416047282s"
Nov 02 01:14:18.196432 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:18.195690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:18.196432 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:18.195984    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:18.196432 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.196196    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:18.196432 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.196259    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:22.196245057 +0000 UTC m=+9.117370360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:18.196432 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.196308    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:18.196432 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.196384    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret podName:7b6da542-d39f-4a29-9849-08164ec387e0 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:22.196324147 +0000 UTC m=+9.117449450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret") pod "global-pull-secret-syncer-nm87s" (UID: "7b6da542-d39f-4a29-9849-08164ec387e0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:18.297201 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:18.297118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:18.297421 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.297400    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:14:18.297502 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.297433    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:14:18.297502 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.297447    2569 projected.go:196] Error preparing data for projected volume kube-api-access-6bq9l for pod openshift-network-diagnostics/network-check-target-tkzsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:18.297609 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.297533    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l podName:12cc7107-489a-4616-a157-e6321337b9be nodeName:}" failed. No retries permitted until 2026-11-02 01:14:22.297511728 +0000 UTC m=+9.218637050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bq9l" (UniqueName: "kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l") pod "network-check-target-tkzsw" (UID: "12cc7107-489a-4616-a157-e6321337b9be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:18.533362 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:18.533330    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:18.533555 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:18.533377    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:18.533555 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:18.533338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:18.533555 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.533463    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:18.533722 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.533613    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:18.533779 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:18.533762    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:19.713165 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:19.713129    2569 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="7e50317c6a7f7ea04763340b1ed4701406149783494e211ddb142e3523d20dee" exitCode=0
Nov 02 01:14:19.713594 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:19.713211    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bzl" event={"ID":"3ad10cee-ccd8-424d-8e5b-8eed99b3881e","Type":"ContainerDied","Data":"7e50317c6a7f7ea04763340b1ed4701406149783494e211ddb142e3523d20dee"}
Nov 02 01:14:19.729130 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:19.729095    2569 generic.go:356] "Generic (PLEG): container finished" podID="699c49bf3206a78424a5f5d5e4f1b0bc" containerID="7416ae5fe75b9d1ed86dd4570e3ac5ba4783e9b55750b54adfa6d7462b26e5ab" exitCode=0
Nov 02 01:14:19.729311 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:19.729160    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal" event={"ID":"699c49bf3206a78424a5f5d5e4f1b0bc","Type":"ContainerDied","Data":"7416ae5fe75b9d1ed86dd4570e3ac5ba4783e9b55750b54adfa6d7462b26e5ab"}
Nov 02 01:14:20.532854 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:20.532771    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:20.532854 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:20.532821    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:20.533078 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:20.532771    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:20.533078 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:20.532933    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:20.533078 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:20.533035    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:20.533237 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:20.533131    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:20.738335 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:20.738296    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dqdrz" event={"ID":"96b08fc5-cc99-4228-9a22-1e57c7fed18b","Type":"ContainerStarted","Data":"495ecbc059d747304e3cd61a33b35918b7e387fcd74ba5f143d4ac765c9f6277"}
Nov 02 01:14:20.747904 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:20.747388    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal" event={"ID":"a10306333dfa6bc03e77a9c914cffe89","Type":"ContainerStarted","Data":"fd49aa6ce592ff5eeb7c7339ffd380b442507a1d83d883321a7e08e96c36620d"}
Nov 02 01:14:20.759107 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:20.757988    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dqdrz" podStartSLOduration=2.082114384 podStartE2EDuration="6.757951747s" totalImagesPullingTime="4.675837363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:20.757160012 +0000 UTC m=+7.678285343" watchObservedRunningTime="2026-11-02 01:14:20.757951747 +0000 UTC m=+7.679077077"
Nov 02 01:14:20.760248 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:20.760030    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal" event={"ID":"699c49bf3206a78424a5f5d5e4f1b0bc","Type":"ContainerStarted","Data":"0268975cb41856a260fbe92a88124dc6cb7b457d322c2d70428330acdd4df59d"}
Nov 02 01:14:20.774510 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:20.774458    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal" podStartSLOduration=6.774439833 podStartE2EDuration="6.774439833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:20.773231968 +0000 UTC m=+7.694357332" watchObservedRunningTime="2026-11-02 01:14:20.774439833 +0000 UTC m=+7.695565182"
Nov 02 01:14:20.791203 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:20.791150    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal" podStartSLOduration=5.791131696 podStartE2EDuration="5.791131696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:20.790566786 +0000 UTC m=+7.711692115" watchObservedRunningTime="2026-11-02 01:14:20.791131696 +0000 UTC m=+7.712257024"
Nov 02 01:14:21.770158 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:21.770106    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlw47" event={"ID":"c4afb19d-ffe5-4011-81a0-5761db34bd75","Type":"ContainerStarted","Data":"0807c200911ac6cfe7e6f87889ed655a7301948b6d65778810f22f77f96b7c05"}
Nov 02 01:14:21.787520 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:21.787457    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hlw47" podStartSLOduration=2.020124316 podStartE2EDuration="7.78743631s" totalImagesPullingTime="5.767311994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:21.78680508 +0000 UTC m=+8.707930410" watchObservedRunningTime="2026-11-02 01:14:21.78743631 +0000 UTC m=+8.708561638"
Nov 02 01:14:22.235266 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:22.234491    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:22.235266 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:22.234544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:22.235266 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.234700    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:22.235266 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.234762    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:30.234740505 +0000 UTC m=+17.155865828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:22.235266 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.235176    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:22.235266 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.235231    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret podName:7b6da542-d39f-4a29-9849-08164ec387e0 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:30.235213634 +0000 UTC m=+17.156338954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret") pod "global-pull-secret-syncer-nm87s" (UID: "7b6da542-d39f-4a29-9849-08164ec387e0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:22.336069 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:22.335290    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:22.336069 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.335561    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:14:22.336069 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.335583    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:14:22.336069 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.335596    2569 projected.go:196] Error preparing data for projected volume kube-api-access-6bq9l for pod openshift-network-diagnostics/network-check-target-tkzsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:22.336069 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.335650    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l podName:12cc7107-489a-4616-a157-e6321337b9be nodeName:}" failed. No retries permitted until 2026-11-02 01:14:30.335636123 +0000 UTC m=+17.256761442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bq9l" (UniqueName: "kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l") pod "network-check-target-tkzsw" (UID: "12cc7107-489a-4616-a157-e6321337b9be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:22.533723 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:22.533347    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:22.533723 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.533452    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:22.533723 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:22.533518    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:22.533723 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.533639    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:22.534191 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:22.534037    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:22.534191 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:22.534129    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:22.773807 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:22.773721    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh" event={"ID":"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83","Type":"ContainerStarted","Data":"1e96f0e9ed493eab22807b93ee43540a03476b6dd571354e74f2398089d4dfda"}
Nov 02 01:14:23.579623 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:23.579597    2569 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 01:14:23.778216 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:23.777959    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fps5b" event={"ID":"eef42078-d140-4216-8ee9-0cc6f0809a33","Type":"ContainerStarted","Data":"b0c834fde1af03f3d221609a527533b7dfab71ded9b07c4b1be3046d3c74f44e"}
Nov 02 01:14:23.780436 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:23.780409    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh" event={"ID":"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83","Type":"ContainerStarted","Data":"e1ae3781b4127f376955199c24c5a96724964841851f3577af828324da08f89c"}
Nov 02 01:14:23.799085 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:23.798884    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-fps5b" podStartSLOduration=1.3452972060000001 podStartE2EDuration="9.79886899s" totalImagesPullingTime="8.453571784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:23.7980891 +0000 UTC m=+10.719214423" watchObservedRunningTime="2026-11-02 01:14:23.79886899 +0000 UTC m=+10.719994507"
Nov 02 01:14:24.533935 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:24.533352    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:24.533935 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:24.533424    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:24.533935 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:24.533443    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:24.533935 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:24.533563    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:24.533935 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:24.533595    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:24.533935 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:24.533655    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:24.535310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:24.535186    2569 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:14:23.579620666Z","UUID":"d7ddab26-8fec-4aeb-b21e-cd51dfbc5f4d","Handler":null,"Name":"","Endpoint":""}
Nov 02 01:14:24.537413 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:24.537389    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 01:14:24.537413 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:24.537418    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 01:14:24.784463 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:24.784389    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2wxd5" event={"ID":"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f","Type":"ContainerStarted","Data":"7d98c0eac4b3a86b5ae5b9c08ac97c484e038b5c3f8ffe139917cc5db5558105"}
Nov 02 01:14:24.788316 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:24.787889    2569 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="d31a5eaf3ae7afdb6f7732811dc4863a97a2acc97719f4bdde020087c9479015" exitCode=0
Nov 02 01:14:24.788316 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:24.787991    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bzl" event={"ID":"3ad10cee-ccd8-424d-8e5b-8eed99b3881e","Type":"ContainerDied","Data":"d31a5eaf3ae7afdb6f7732811dc4863a97a2acc97719f4bdde020087c9479015"}
Nov 02 01:14:24.829774 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:24.829706    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2wxd5" podStartSLOduration=1.459075642 podStartE2EDuration="10.82968721s" totalImagesPullingTime="9.370611568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:24.801407409 +0000 UTC m=+11.722532733" watchObservedRunningTime="2026-11-02 01:14:24.82968721 +0000 UTC m=+11.750812535"
Nov 02 01:14:25.445924 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:25.445860    2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:25.446717 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:25.446696    2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:25.791559 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:25.791486    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nk8xc" event={"ID":"ce7ea8b0-06fd-457e-804b-da8c4f73edfc","Type":"ContainerStarted","Data":"3ff5704cc866befc2d31980e30ca4530e4400c03a8cc1295f961852fb9199d2d"}
Nov 02 01:14:25.793546 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:25.793523    2569 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="f0c3add76a70492a448c40342f67958665cd0873310e70ffc025edc98782e6c5" exitCode=0
Nov 02 01:14:25.793646 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:25.793593    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bzl" event={"ID":"3ad10cee-ccd8-424d-8e5b-8eed99b3881e","Type":"ContainerDied","Data":"f0c3add76a70492a448c40342f67958665cd0873310e70ffc025edc98782e6c5"}
Nov 02 01:14:25.795968 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:25.795924    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh" event={"ID":"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83","Type":"ContainerStarted","Data":"a64d65a6f49ca00d4d92122106d3696dec631f6392bd2263f4224427101d3414"}
Nov 02 01:14:25.807389 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:25.807336    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nk8xc" podStartSLOduration=2.906973018 podStartE2EDuration="11.807322044s" totalImagesPullingTime="8.900349026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:25.806772609 +0000 UTC m=+12.727897934" watchObservedRunningTime="2026-11-02 01:14:25.807322044 +0000 UTC m=+12.728447382"
Nov 02 01:14:25.826180 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:25.826130    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh" podStartSLOduration=2.017073659 podStartE2EDuration="11.826117024s" totalImagesPullingTime="9.809043365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:25.825965086 +0000 UTC m=+12.747090410" watchObservedRunningTime="2026-11-02 01:14:25.826117024 +0000 UTC m=+12.747242396"
Nov 02 01:14:26.533540 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:26.533511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:26.533540 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:26.533531    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:26.533735 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:26.533521    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:26.533735 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:26.533645    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:26.533858 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:26.533747    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:26.533858 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:26.533806    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:26.800247 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:26.800212    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bzl" event={"ID":"3ad10cee-ccd8-424d-8e5b-8eed99b3881e","Type":"ContainerStarted","Data":"f977bc985c216d036978e343c6b779dfa7a0709aedcd5f7ca14c730f9f283a7c"}
Nov 02 01:14:26.802160 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:26.802136    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r2dhn" event={"ID":"5d45205b-f684-4607-b594-0352c721a4f7","Type":"ContainerStarted","Data":"648ef8bc43dda2b9963483f6617f02fb9ee79752d8ee57ab5f709fc0d33b69b4"}
Nov 02 01:14:26.845886 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:26.845843    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r2dhn" podStartSLOduration=1.7938016060000002 podStartE2EDuration="12.845829889s" totalImagesPullingTime="11.052028283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:26.845163164 +0000 UTC m=+13.766288487" watchObservedRunningTime="2026-11-02 01:14:26.845829889 +0000 UTC m=+13.766955213"
Nov 02 01:14:27.808198 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:27.807941    2569 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="f977bc985c216d036978e343c6b779dfa7a0709aedcd5f7ca14c730f9f283a7c" exitCode=0
Nov 02 01:14:27.808689 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:27.808002    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bzl" event={"ID":"3ad10cee-ccd8-424d-8e5b-8eed99b3881e","Type":"ContainerDied","Data":"f977bc985c216d036978e343c6b779dfa7a0709aedcd5f7ca14c730f9f283a7c"}
Nov 02 01:14:28.533538 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:28.533504    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:28.533760 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:28.533571    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:28.533760 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:28.533568    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:28.533760 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:28.533673    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:28.533961 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:28.533764    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:28.533961 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:28.533851    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:30.296640 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:30.296567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:30.296640 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:30.296607    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:30.297118 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.296724    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:30.297118 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.296724    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:30.297118 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.296789    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret podName:7b6da542-d39f-4a29-9849-08164ec387e0 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:46.296775138 +0000 UTC m=+33.217900457 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret") pod "global-pull-secret-syncer-nm87s" (UID: "7b6da542-d39f-4a29-9849-08164ec387e0") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:14:30.297118 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.296804    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:46.29679711 +0000 UTC m=+33.217922412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:14:30.397361 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:30.397330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:30.397536 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.397478    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:14:30.397536 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.397496    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:14:30.397536 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.397504    2569 projected.go:196] Error preparing data for projected volume kube-api-access-6bq9l for pod openshift-network-diagnostics/network-check-target-tkzsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:30.397671 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.397561    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l podName:12cc7107-489a-4616-a157-e6321337b9be nodeName:}" failed. No retries permitted until 2026-11-02 01:14:46.397544092 +0000 UTC m=+33.318669414 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bq9l" (UniqueName: "kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l") pod "network-check-target-tkzsw" (UID: "12cc7107-489a-4616-a157-e6321337b9be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:14:30.532622 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:30.532587    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:30.532794 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:30.532590    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:30.532794 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.532707    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:30.532899 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.532787    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:30.532899 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:30.532595    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:30.532899 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:30.532856    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:30.816790 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:30.816759    2569 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="1a74d5036266f00e100f60931147831c75cf4718f91ca24ba4aa0c3e12eacc5c" exitCode=0
Nov 02 01:14:30.816930 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:30.816817    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bzl" event={"ID":"3ad10cee-ccd8-424d-8e5b-8eed99b3881e","Type":"ContainerDied","Data":"1a74d5036266f00e100f60931147831c75cf4718f91ca24ba4aa0c3e12eacc5c"}
Nov 02 01:14:31.821844 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:31.821609    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6" event={"ID":"68a83bf9-59f7-42a3-aeb0-efc8691a6644","Type":"ContainerStarted","Data":"0fcda6f7206857f3c5fe73fde6a9d9f27b1dc2ef47ade7886862d69a18f24f94"}
Nov 02 01:14:31.821844 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:31.821830    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6" event={"ID":"68a83bf9-59f7-42a3-aeb0-efc8691a6644","Type":"ContainerStarted","Data":"20454b090916112873899159622a17be9f743c2613c60dedef525081bbea1f54"}
Nov 02 01:14:31.822419 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:31.821878    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6" event={"ID":"68a83bf9-59f7-42a3-aeb0-efc8691a6644","Type":"ContainerStarted","Data":"51f339b6480fea5bdaa8a6a7f6067207c96645bd92bf5702786e9eeedace7828"}
Nov 02 01:14:31.822419 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:31.821908    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6" event={"ID":"68a83bf9-59f7-42a3-aeb0-efc8691a6644","Type":"ContainerStarted","Data":"514bb7bf9bfa9e9d70d5f12e3101c22b89944504e2de3c7dd04786620be34ec6"}
Nov 02 01:14:31.822419 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:31.821935    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6" event={"ID":"68a83bf9-59f7-42a3-aeb0-efc8691a6644","Type":"ContainerStarted","Data":"e4525a7ea3b42bfbc64c9efe155a94cf3fb68cace78727083bd1e7ff4be7e55d"}
Nov 02 01:14:31.822419 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:31.821962    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6" event={"ID":"68a83bf9-59f7-42a3-aeb0-efc8691a6644","Type":"ContainerStarted","Data":"f0e56dd5f3fdb0091b37ad2834a752c182da103eeec7335e17b665f341cfb389"}
Nov 02 01:14:31.824523 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:31.824498    2569 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="05650d2d86fa9422b4ae0f444ab8f0ddfd8446daeede737d1871a0d2dcb4b352" exitCode=0
Nov 02 01:14:31.824642 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:31.824544    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bzl" event={"ID":"3ad10cee-ccd8-424d-8e5b-8eed99b3881e","Type":"ContainerDied","Data":"05650d2d86fa9422b4ae0f444ab8f0ddfd8446daeede737d1871a0d2dcb4b352"}
Nov 02 01:14:32.533177 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:32.533139    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:32.533177 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:32.533161    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:32.533419 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:32.533154    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:32.533419 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:32.533238    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:32.533419 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:32.533344    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:32.533549 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:32.533430    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:32.829534 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:32.829452    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bzl" event={"ID":"3ad10cee-ccd8-424d-8e5b-8eed99b3881e","Type":"ContainerStarted","Data":"37ec5e1cf0ebb1d550489b162e205ce8a5f2747a5ed29f796c9f17636e311a39"}
Nov 02 01:14:32.855938 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:32.855897    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z9bzl" podStartSLOduration=7.55849549 podStartE2EDuration="18.855885405s" totalImagesPullingTime="11.297389915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:32.855462342 +0000 UTC m=+19.776587667" watchObservedRunningTime="2026-11-02 01:14:32.855885405 +0000 UTC m=+19.777010728"
Nov 02 01:14:33.835108 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:33.835031    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6" event={"ID":"68a83bf9-59f7-42a3-aeb0-efc8691a6644","Type":"ContainerStarted","Data":"959f4cec1ed0250e8c3af76d525d8eda56381a694b4619f1de00cbef10580995"}
Nov 02 01:14:34.532895 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:34.532863    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:34.533048 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:34.532980    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:34.533048 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:34.532991    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:34.533048 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:34.532989    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:34.533147 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:34.533077    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:34.533177 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:34.533150    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:35.841632 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:35.841554    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6" event={"ID":"68a83bf9-59f7-42a3-aeb0-efc8691a6644","Type":"ContainerStarted","Data":"25b0e05211bbfccbdb6e00b24e6356e0ad94ab3b569526728401132cb588b597"}
Nov 02 01:14:35.842225 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:35.841797    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:35.842225 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:35.841889    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:35.857637 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:35.857486    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:35.875356 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:35.875315    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6" podStartSLOduration=5.656932283 podStartE2EDuration="21.875283646s" totalImagesPullingTime="16.218351363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:35.873682638 +0000 UTC m=+22.794807961" watchObservedRunningTime="2026-11-02 01:14:35.875283646 +0000 UTC m=+22.796408990"
Nov 02 01:14:36.533446 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:36.533412    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:36.533723 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:36.533415    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:36.533723 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:36.533511    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:36.533723 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:36.533520    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:36.533723 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:36.533605    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:36.533723 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:36.533642    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:36.844522 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:36.844401    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:36.861194 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:36.861166    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:37.340517 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:37.340446    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nm87s"]
Nov 02 01:14:37.340664 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:37.340582    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:37.340710 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:37.340686    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:37.347460 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:37.347437    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tkzsw"]
Nov 02 01:14:37.347598 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:37.347539    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:37.347666 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:37.347629    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:37.350110 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:37.350093    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2hkrs"]
Nov 02 01:14:37.350206 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:37.350170    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:37.350285 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:37.350265    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:39.532729 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:39.532656    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:39.532729 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:39.532692    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:39.533090 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:39.532663    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:39.533090 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:39.532772    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:39.533090 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:39.532836    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:39.533090 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:39.532905    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:40.521063 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:40.520968    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:40.521239 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:40.521156    2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:14:40.521602 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:40.521570    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:41.532778 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:41.532749    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:41.532778 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:41.532773    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:41.533330 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:41.532838    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:41.533330 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:41.532880    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:41.533330 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:41.532948    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:41.533330 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:41.533029    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:43.533378 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.533344    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:43.533835 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:43.533444    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nm87s" podUID="7b6da542-d39f-4a29-9849-08164ec387e0"
Nov 02 01:14:43.533835 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.533487    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:43.533835 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.533607    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:43.533835 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:43.533604    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tkzsw" podUID="12cc7107-489a-4616-a157-e6321337b9be"
Nov 02 01:14:43.533835 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:43.533730    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hkrs" podUID="0ccf76ab-a4c9-4087-8ada-59b4c19218f8"
Nov 02 01:14:43.851257 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.851191    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeReady"
Nov 02 01:14:43.851418 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.851381    2569 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 01:14:43.887951 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.887925    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p9c2l"]
Nov 02 01:14:43.895054 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.895025    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5jpp9"]
Nov 02 01:14:43.896172 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.895590    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-p9c2l"
Nov 02 01:14:43.897384 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.897365    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jhbp5"]
Nov 02 01:14:43.898245 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.898223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5jpp9"
Nov 02 01:14:43.899762 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.899741    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7978dd58c5-947pf"]
Nov 02 01:14:43.900545 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.900523    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jhbp5"
Nov 02 01:14:43.901929 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.901904    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"]
Nov 02 01:14:43.902026 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.902007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:43.904485 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.904470    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"]
Nov 02 01:14:43.904587 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.904574    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:43.906655 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.906630    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bdwzr"]
Nov 02 01:14:43.906758 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.906744    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:43.910968 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.910949    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv"]
Nov 02 01:14:43.911056 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.911016    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:43.911252 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.911230    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 01:14:43.911524 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.911508    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 01:14:43.912229 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.912209    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 01:14:43.912353 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.912317    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2zfq9\""
Nov 02 01:14:43.912505 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.912487    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 01:14:43.915066 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.915046    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 01:14:43.915842 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.915817    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"]
Nov 02 01:14:43.916155 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.916137    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv"
Nov 02 01:14:43.920081 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.920057    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 01:14:43.923566 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.923016    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c986fbdbf-f69k6"]
Nov 02 01:14:43.924338 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.924164    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:43.926622 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.926605    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"]
Nov 02 01:14:43.926808 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.926788    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:43.929933 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.929916    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"]
Nov 02 01:14:43.930050 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.930034    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:43.932684 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.932665    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76d8c5d8fd-g46xp"]
Nov 02 01:14:43.933077 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.933059    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:43.934835 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.934820    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"]
Nov 02 01:14:43.934955 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.934941    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:43.944569 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.944494    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68"]
Nov 02 01:14:43.944569 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.944562    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:43.948278 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.948092    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5jpp9"]
Nov 02 01:14:43.948278 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.948133    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p9c2l"]
Nov 02 01:14:43.948278 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.948156    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7zmf4"]
Nov 02 01:14:43.948559 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.948542    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68"
Nov 02 01:14:43.952366 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.952346    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jhbp5"]
Nov 02 01:14:43.952479 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.952383    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"]
Nov 02 01:14:43.952479 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.952407    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wlvr8"]
Nov 02 01:14:43.952479 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.952467    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:43.954982 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.954967    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"]
Nov 02 01:14:43.955067 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.954994    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"]
Nov 02 01:14:43.955067 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955012    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"]
Nov 02 01:14:43.955067 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955042    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68"]
Nov 02 01:14:43.955067 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955062    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7978dd58c5-947pf"]
Nov 02 01:14:43.955258 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955070    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:43.955258 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955083    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bdwzr"]
Nov 02 01:14:43.955258 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955103    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7zmf4"]
Nov 02 01:14:43.955258 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955127    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv"]
Nov 02 01:14:43.955258 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955153    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"]
Nov 02 01:14:43.955258 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955173    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76d8c5d8fd-g46xp"]
Nov 02 01:14:43.955258 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955193    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"]
Nov 02 01:14:43.955258 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.955213    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c986fbdbf-f69k6"]
Nov 02 01:14:43.959339 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.959317    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wlvr8"]
Nov 02 01:14:43.998830 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.998801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-tmp\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:43.998967 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.998859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-image-registry-private-configuration\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:43.999029 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.998976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/414ef228-1302-4af5-8118-be23a4280494-serving-cert\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:43.999029 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999007    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkffz\" (UniqueName: \"kubernetes.io/projected/b395656d-d1a0-4cea-9b61-41788ab06732-kube-api-access-pkffz\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:43.999029 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999027    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5krs\" (UniqueName: \"kubernetes.io/projected/414ef228-1302-4af5-8118-be23a4280494-kube-api-access-w5krs\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:43.999154 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7wxp\" (UniqueName: \"kubernetes.io/projected/50991da9-4137-467f-a695-0d0d4203321f-kube-api-access-j7wxp\") pod \"kas-connection-checker-685895c7c9-5jpp9\" (UID: \"50991da9-4137-467f-a695-0d0d4203321f\") " pod="kube-system/kas-connection-checker-685895c7c9-5jpp9"
Nov 02 01:14:43.999154 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999134    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:43.999255 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999206    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:43.999255 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:43.999377 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999264    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1b792a4-f821-4dec-9db4-36bfeceee2e6-ca-trust-extracted\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:43.999377 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999326    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:43.999470 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-installation-pull-secrets\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:43.999470 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999395    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg6v7\" (UniqueName: \"kubernetes.io/projected/16b92078-b559-44f6-85b0-4f9d1da9ddcf-kube-api-access-sg6v7\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:43.999470 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999415    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:43.999470 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999429    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-trusted-ca\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:43.999658 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999478    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-service-ca-bundle\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:43.999658 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999507    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16b92078-b559-44f6-85b0-4f9d1da9ddcf-serving-cert\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:43.999658 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999535    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvkf2\" (UniqueName: \"kubernetes.io/projected/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-kube-api-access-vvkf2\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:43.999658 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999616    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-stats-auth\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:43.999825 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999666    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:43.999825 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999695    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-snapshots\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:43.999825 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e57114c7-9426-4d83-870a-9d033cf335e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:43.999825 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999771    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-default-certificate\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:43.999825 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999796    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-trusted-ca\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:43.999825 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc662357-7bcd-4efb-9857-37a197cb67bf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:44.000087 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:44.000087 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999860    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/414ef228-1302-4af5-8118-be23a4280494-config\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:44.000087 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-certificates\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.000087 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:43.999945    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmk5\" (UniqueName: \"kubernetes.io/projected/91f2617a-9008-4f4a-ada4-8d5997064b52-kube-api-access-sbmk5\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:44.000087 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.000050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7h5m\" (UniqueName: \"kubernetes.io/projected/e57114c7-9426-4d83-870a-9d033cf335e4-kube-api-access-f7h5m\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:44.000279 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.000091    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87d7t\" (UniqueName: \"kubernetes.io/projected/cc662357-7bcd-4efb-9857-37a197cb67bf-kube-api-access-87d7t\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:44.000279 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.000144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrqr\" (UniqueName: \"kubernetes.io/projected/8d515403-65e0-4198-b78a-794ef628bc8f-kube-api-access-jkrqr\") pod \"kas-connection-checker-685895c7c9-jhbp5\" (UID: \"8d515403-65e0-4198-b78a-794ef628bc8f\") " pod="kube-system/kas-connection-checker-685895c7c9-jhbp5"
Nov 02 01:14:44.000279 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.000179    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-bound-sa-token\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.000279 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.000207    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-config\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.000554 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.000359    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc662357-7bcd-4efb-9857-37a197cb67bf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:44.000554 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.000386    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b395656d-d1a0-4cea-9b61-41788ab06732-serving-cert\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.000554 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.000411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpxv\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-kube-api-access-bfpxv\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.000554 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.000436    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pqb4\" (UniqueName: \"kubernetes.io/projected/4b7adf12-c300-4019-8425-6115d6ebca69-kube-api-access-2pqb4\") pod \"volume-data-source-validator-58c9f5f46d-b4llv\" (UID: \"4b7adf12-c300-4019-8425-6115d6ebca69\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv"
Nov 02 01:14:44.000554 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.000463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd26r\" (UniqueName: \"kubernetes.io/projected/975d3545-32e4-4d97-a759-d4251bb28224-kube-api-access-bd26r\") pod \"kas-connection-checker-685895c7c9-p9c2l\" (UID: \"975d3545-32e4-4d97-a759-d4251bb28224\") " pod="kube-system/kas-connection-checker-685895c7c9-p9c2l"
Nov 02 01:14:44.011868 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.011845    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:44.011958 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.011934    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 01:14:44.011958 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.011939    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 01:14:44.012080 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.011978    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 01:14:44.012080 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012025    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 01:14:44.012080 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.011992    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:44.012276 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012080    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 01:14:44.012276 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012086    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 01:14:44.012276 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012006    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 01:14:44.012276 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012150    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 01:14:44.012276 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012174    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:14:44.012276 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012231    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 01:14:44.012276 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012236    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 01:14:44.012276 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012240    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 01:14:44.012703 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012311    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:44.012703 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012321    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 01:14:44.012703 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012578    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 01:14:44.012703 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012583    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 01:14:44.012703 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012655    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:44.012703 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012657    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wsv4x\""
Nov 02 01:14:44.012703 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012676    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:14:44.012703 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012682    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:44.012980 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012746    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 01:14:44.012980 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.012819    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 01:14:44.013106 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013080    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 01:14:44.013106 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013103    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 01:14:44.013209 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013085    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 01:14:44.013209 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013154    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 01:14:44.013209 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013179    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 01:14:44.013209 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013196    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:44.013417 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013236    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 01:14:44.013417 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013237    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:44.013417 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013255    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 01:14:44.013417 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013256    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 01:14:44.013417 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013274    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:44.013417 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013305    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:44.013417 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013337    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:44.013417 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013340    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 01:14:44.013417 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013391    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 01:14:44.013939 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.013925    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 01:14:44.014072 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014059    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-btsp4\""
Nov 02 01:14:44.014126 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014105    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 01:14:44.014255 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014240    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7rblb\""
Nov 02 01:14:44.014337 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014259    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qj2bz\""
Nov 02 01:14:44.014337 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014266    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-lq78q\""
Nov 02 01:14:44.014337 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014263    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rcvqj\""
Nov 02 01:14:44.014337 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014325    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jbz7p\""
Nov 02 01:14:44.014476 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014331    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6rtc\""
Nov 02 01:14:44.014476 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014337    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bjjp6\""
Nov 02 01:14:44.014476 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014372    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pm2nf\""
Nov 02 01:14:44.014476 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.014443    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zb852\""
Nov 02 01:14:44.016185 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.016167    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 01:14:44.016260 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.016168    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8sg8n\""
Nov 02 01:14:44.019468 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.019441    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 01:14:44.019616 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.019597    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 01:14:44.101705 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc662357-7bcd-4efb-9857-37a197cb67bf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:44.101705 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101681    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b395656d-d1a0-4cea-9b61-41788ab06732-serving-cert\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.101902 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101719    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpxv\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-kube-api-access-bfpxv\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.101902 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pqb4\" (UniqueName: \"kubernetes.io/projected/4b7adf12-c300-4019-8425-6115d6ebca69-kube-api-access-2pqb4\") pod \"volume-data-source-validator-58c9f5f46d-b4llv\" (UID: \"4b7adf12-c300-4019-8425-6115d6ebca69\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv"
Nov 02 01:14:44.101902 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101779    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-image-registry-private-configuration\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.101902 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101812    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.101902 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101846    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd26r\" (UniqueName: \"kubernetes.io/projected/975d3545-32e4-4d97-a759-d4251bb28224-kube-api-access-bd26r\") pod \"kas-connection-checker-685895c7c9-p9c2l\" (UID: \"975d3545-32e4-4d97-a759-d4251bb28224\") " pod="kube-system/kas-connection-checker-685895c7c9-p9c2l"
Nov 02 01:14:44.101902 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101879    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-installation-pull-secrets\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.102152 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101910    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-tmp\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.102152 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wxr\" (UniqueName: \"kubernetes.io/projected/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-kube-api-access-27wxr\") pod \"ingress-canary-wlvr8\" (UID: \"51025d9a-739c-4934-9f5b-5bcbeb0d81ca\") " pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:44.102152 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.101972    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be28aae-f92c-4eb1-b063-f84b46967f3b-tmp-dir\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.102152 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102017    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-image-registry-private-configuration\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.102152 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102049    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-certificates\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.102152 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102083    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/414ef228-1302-4af5-8118-be23a4280494-serving-cert\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:44.102152 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkffz\" (UniqueName: \"kubernetes.io/projected/b395656d-d1a0-4cea-9b61-41788ab06732-kube-api-access-pkffz\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.102152 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-trusted-ca\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.102529 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102167    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-cert\") pod \"ingress-canary-wlvr8\" (UID: \"51025d9a-739c-4934-9f5b-5bcbeb0d81ca\") " pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:44.102529 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5krs\" (UniqueName: \"kubernetes.io/projected/414ef228-1302-4af5-8118-be23a4280494-kube-api-access-w5krs\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:44.102529 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102245    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wxp\" (UniqueName: \"kubernetes.io/projected/50991da9-4137-467f-a695-0d0d4203321f-kube-api-access-j7wxp\") pod \"kas-connection-checker-685895c7c9-5jpp9\" (UID: \"50991da9-4137-467f-a695-0d0d4203321f\") " pod="kube-system/kas-connection-checker-685895c7c9-5jpp9"
Nov 02 01:14:44.102529 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102274    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.102529 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:44.102529 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.102529 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102405    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1b792a4-f821-4dec-9db4-36bfeceee2e6-ca-trust-extracted\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.102529 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:44.102529 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102524    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1be28aae-f92c-4eb1-b063-f84b46967f3b-config-volume\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.102915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102562    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-installation-pull-secrets\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.102915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg6v7\" (UniqueName: \"kubernetes.io/projected/16b92078-b559-44f6-85b0-4f9d1da9ddcf-kube-api-access-sg6v7\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.102915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.102915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:44.102915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-trusted-ca\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.102915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-service-ca-bundle\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.102915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102777    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16b92078-b559-44f6-85b0-4f9d1da9ddcf-serving-cert\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.102915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkf2\" (UniqueName: \"kubernetes.io/projected/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-kube-api-access-vvkf2\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:44.102915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-stats-auth\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:44.102915 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102902    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn6dd\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-kube-api-access-bn6dd\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.102965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-snapshots\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103004    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103035    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e57114c7-9426-4d83-870a-9d033cf335e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-default-certificate\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fp48\" (UniqueName: \"kubernetes.io/projected/1be28aae-f92c-4eb1-b063-f84b46967f3b-kube-api-access-5fp48\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103190    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-trusted-ca\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103218    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc662357-7bcd-4efb-9857-37a197cb67bf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103242    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103271    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qld9n\" (UniqueName: \"kubernetes.io/projected/9fb03996-eac9-4517-9b19-283115a3ba00-kube-api-access-qld9n\") pod \"network-check-source-6f78d59f99-cjb68\" (UID: \"9fb03996-eac9-4517-9b19-283115a3ba00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103314    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb996d25-e713-4ed5-a1db-871234d5a8cd-ca-trust-extracted\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.103388 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/414ef228-1302-4af5-8118-be23a4280494-config\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:44.103939 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103409    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-certificates\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.103939 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2edcda9-40c1-4565-8843-1083f0f60b5a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:44.103939 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmk5\" (UniqueName: \"kubernetes.io/projected/91f2617a-9008-4f4a-ada4-8d5997064b52-kube-api-access-sbmk5\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:44.103939 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103536    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7h5m\" (UniqueName: \"kubernetes.io/projected/e57114c7-9426-4d83-870a-9d033cf335e4-kube-api-access-f7h5m\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:44.103939 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87d7t\" (UniqueName: \"kubernetes.io/projected/cc662357-7bcd-4efb-9857-37a197cb67bf-kube-api-access-87d7t\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:44.103939 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrqr\" (UniqueName: \"kubernetes.io/projected/8d515403-65e0-4198-b78a-794ef628bc8f-kube-api-access-jkrqr\") pod \"kas-connection-checker-685895c7c9-jhbp5\" (UID: \"8d515403-65e0-4198-b78a-794ef628bc8f\") " pod="kube-system/kas-connection-checker-685895c7c9-jhbp5"
Nov 02 01:14:44.103939 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-bound-sa-token\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.103939 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103674    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-bound-sa-token\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.103939 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.103705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-config\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.104530 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.104507    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-config\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.105102 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.105078    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:14:44.105187 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.105158    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:44.605138354 +0000 UTC m=+31.526263660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : secret "router-metrics-certs-default" not found
Nov 02 01:14:44.105736 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.105708    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1b792a4-f821-4dec-9db4-36bfeceee2e6-ca-trust-extracted\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.105845 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.105752    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:44.105845 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.105769    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7978dd58c5-947pf: secret "image-registry-tls" not found
Nov 02 01:14:44.105845 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.105837    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls podName:d1b792a4-f821-4dec-9db4-36bfeceee2e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:44.605813959 +0000 UTC m=+31.526939279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls") pod "image-registry-7978dd58c5-947pf" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6") : secret "image-registry-tls" not found
Nov 02 01:14:44.106125 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.106103    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:44.606086074 +0000 UTC m=+31.527211398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : configmap references non-existent config key: service-ca.crt
Nov 02 01:14:44.106219 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.106166    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/414ef228-1302-4af5-8118-be23a4280494-config\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:44.106329 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.106282    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-certificates\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.106430 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.106410    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:44.107420 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.107347    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e57114c7-9426-4d83-870a-9d033cf335e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:44.107518 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.107396    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.107600 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.107581    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:14:44.107695 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.107682    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls podName:91f2617a-9008-4f4a-ada4-8d5997064b52 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:44.607663632 +0000 UTC m=+31.528788938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2v57l" (UID: "91f2617a-9008-4f4a-ada4-8d5997064b52") : secret "samples-operator-tls" not found
Nov 02 01:14:44.107886 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.107863    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc662357-7bcd-4efb-9857-37a197cb67bf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:44.108004 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.107979    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-image-registry-private-configuration\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.108064 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.108028    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-service-ca-bundle\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.108231 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.108202    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-trusted-ca\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.108589 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.108565    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-trusted-ca\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.109150 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.109127    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc662357-7bcd-4efb-9857-37a197cb67bf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:44.109233 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.109188    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-installation-pull-secrets\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.109233 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.109215    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/414ef228-1302-4af5-8118-be23a4280494-serving-cert\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:44.109367 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.109266    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:44.109367 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.109348    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls podName:e57114c7-9426-4d83-870a-9d033cf335e4 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:44.609330775 +0000 UTC m=+31.530456081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gsw4b" (UID: "e57114c7-9426-4d83-870a-9d033cf335e4") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:44.109514 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.109451    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-default-certificate\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:44.109589 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.109559    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-snapshots\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.109714 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.109501    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-tmp\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.110886 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.110859    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b395656d-d1a0-4cea-9b61-41788ab06732-serving-cert\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.111147 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.111131    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-stats-auth\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:44.111617 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.111600    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16b92078-b559-44f6-85b0-4f9d1da9ddcf-serving-cert\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.115209 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.115184    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg6v7\" (UniqueName: \"kubernetes.io/projected/16b92078-b559-44f6-85b0-4f9d1da9ddcf-kube-api-access-sg6v7\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.115691 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.115672    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd26r\" (UniqueName: \"kubernetes.io/projected/975d3545-32e4-4d97-a759-d4251bb28224-kube-api-access-bd26r\") pod \"kas-connection-checker-685895c7c9-p9c2l\" (UID: \"975d3545-32e4-4d97-a759-d4251bb28224\") " pod="kube-system/kas-connection-checker-685895c7c9-p9c2l"
Nov 02 01:14:44.120618 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.120543    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pqb4\" (UniqueName: \"kubernetes.io/projected/4b7adf12-c300-4019-8425-6115d6ebca69-kube-api-access-2pqb4\") pod \"volume-data-source-validator-58c9f5f46d-b4llv\" (UID: \"4b7adf12-c300-4019-8425-6115d6ebca69\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv"
Nov 02 01:14:44.120714 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.120669    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5krs\" (UniqueName: \"kubernetes.io/projected/414ef228-1302-4af5-8118-be23a4280494-kube-api-access-w5krs\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:44.121224 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.121207    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpxv\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-kube-api-access-bfpxv\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.125573 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.125546    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7h5m\" (UniqueName: \"kubernetes.io/projected/e57114c7-9426-4d83-870a-9d033cf335e4-kube-api-access-f7h5m\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:44.127158 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.127141    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkf2\" (UniqueName: \"kubernetes.io/projected/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-kube-api-access-vvkf2\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:44.127525 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.127505    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87d7t\" (UniqueName: \"kubernetes.io/projected/cc662357-7bcd-4efb-9857-37a197cb67bf-kube-api-access-87d7t\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:44.127616 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.127594    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrqr\" (UniqueName: \"kubernetes.io/projected/8d515403-65e0-4198-b78a-794ef628bc8f-kube-api-access-jkrqr\") pod \"kas-connection-checker-685895c7c9-jhbp5\" (UID: \"8d515403-65e0-4198-b78a-794ef628bc8f\") " pod="kube-system/kas-connection-checker-685895c7c9-jhbp5"
Nov 02 01:14:44.127654 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.127621    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wxp\" (UniqueName: \"kubernetes.io/projected/50991da9-4137-467f-a695-0d0d4203321f-kube-api-access-j7wxp\") pod \"kas-connection-checker-685895c7c9-5jpp9\" (UID: \"50991da9-4137-467f-a695-0d0d4203321f\") " pod="kube-system/kas-connection-checker-685895c7c9-5jpp9"
Nov 02 01:14:44.127754 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.127735    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmk5\" (UniqueName: \"kubernetes.io/projected/91f2617a-9008-4f4a-ada4-8d5997064b52-kube-api-access-sbmk5\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:44.127784 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.127754    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkffz\" (UniqueName: \"kubernetes.io/projected/b395656d-d1a0-4cea-9b61-41788ab06732-kube-api-access-pkffz\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.128577 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.128557    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-bound-sa-token\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.214131 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214100    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-p9c2l"
Nov 02 01:14:44.214318 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214280    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2edcda9-40c1-4565-8843-1083f0f60b5a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:44.214393 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-bound-sa-token\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.214393 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214373    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-image-registry-private-configuration\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.214489 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.214489 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214423    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-installation-pull-secrets\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.214489 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214452    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wxr\" (UniqueName: \"kubernetes.io/projected/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-kube-api-access-27wxr\") pod \"ingress-canary-wlvr8\" (UID: \"51025d9a-739c-4934-9f5b-5bcbeb0d81ca\") " pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:44.214489 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be28aae-f92c-4eb1-b063-f84b46967f3b-tmp-dir\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.214666 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214532    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-certificates\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.214666 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.214566    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:14:44.214666 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-trusted-ca\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.214666 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-cert\") pod \"ingress-canary-wlvr8\" (UID: \"51025d9a-739c-4934-9f5b-5bcbeb0d81ca\") " pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:44.214666 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.214618    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls podName:1be28aae-f92c-4eb1-b063-f84b46967f3b nodeName:}" failed. No retries permitted until 2026-11-02 01:14:44.714599729 +0000 UTC m=+31.635725052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls") pod "dns-default-7zmf4" (UID: "1be28aae-f92c-4eb1-b063-f84b46967f3b") : secret "dns-default-metrics-tls" not found
Nov 02 01:14:44.214900 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1be28aae-f92c-4eb1-b063-f84b46967f3b-config-volume\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.214900 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.214900 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.214818    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:44.214900 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.214830    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d8c5d8fd-g46xp: secret "image-registry-tls" not found
Nov 02 01:14:44.214900 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.214866    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls podName:fb996d25-e713-4ed5-a1db-871234d5a8cd nodeName:}" failed. No retries permitted until 2026-11-02 01:14:44.714854552 +0000 UTC m=+31.635979857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls") pod "image-registry-76d8c5d8fd-g46xp" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd") : secret "image-registry-tls" not found
Nov 02 01:14:44.215122 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.214903    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be28aae-f92c-4eb1-b063-f84b46967f3b-tmp-dir\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.215122 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.215001    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2edcda9-40c1-4565-8843-1083f0f60b5a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:44.215485 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.215440    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1be28aae-f92c-4eb1-b063-f84b46967f3b-config-volume\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.215485 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.215477    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn6dd\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-kube-api-access-bn6dd\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.215735 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.215517    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-certificates\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.215735 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.215527    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:44.215735 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.215577    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fp48\" (UniqueName: \"kubernetes.io/projected/1be28aae-f92c-4eb1-b063-f84b46967f3b-kube-api-access-5fp48\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.215735 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.215604    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:14:44.215735 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.215625    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld9n\" (UniqueName: \"kubernetes.io/projected/9fb03996-eac9-4517-9b19-283115a3ba00-kube-api-access-qld9n\") pod \"network-check-source-6f78d59f99-cjb68\" (UID: \"9fb03996-eac9-4517-9b19-283115a3ba00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68"
Nov 02 01:14:44.215735 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.215655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb996d25-e713-4ed5-a1db-871234d5a8cd-ca-trust-extracted\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.215735 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.215734    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert podName:b2edcda9-40c1-4565-8843-1083f0f60b5a nodeName:}" failed. No retries permitted until 2026-11-02 01:14:44.71571723 +0000 UTC m=+31.636842534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9fbg8" (UID: "b2edcda9-40c1-4565-8843-1083f0f60b5a") : secret "networking-console-plugin-cert" not found
Nov 02 01:14:44.216197 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.216028    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb996d25-e713-4ed5-a1db-871234d5a8cd-ca-trust-extracted\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.216197 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.216074    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-trusted-ca\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.217245 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.217221    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-image-registry-private-configuration\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.217707 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.217691    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-installation-pull-secrets\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.217880 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.217863    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-cert\") pod \"ingress-canary-wlvr8\" (UID: \"51025d9a-739c-4934-9f5b-5bcbeb0d81ca\") " pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:44.224526 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.224511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5jpp9"
Nov 02 01:14:44.228712 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.228694    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-bound-sa-token\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.228864 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.228849    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wxr\" (UniqueName: \"kubernetes.io/projected/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-kube-api-access-27wxr\") pod \"ingress-canary-wlvr8\" (UID: \"51025d9a-739c-4934-9f5b-5bcbeb0d81ca\") " pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:44.229609 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.229595    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jhbp5"
Nov 02 01:14:44.232040 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.232019    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld9n\" (UniqueName: \"kubernetes.io/projected/9fb03996-eac9-4517-9b19-283115a3ba00-kube-api-access-qld9n\") pod \"network-check-source-6f78d59f99-cjb68\" (UID: \"9fb03996-eac9-4517-9b19-283115a3ba00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68"
Nov 02 01:14:44.232164 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.232144    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn6dd\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-kube-api-access-bn6dd\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.232411 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.232397    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fp48\" (UniqueName: \"kubernetes.io/projected/1be28aae-f92c-4eb1-b063-f84b46967f3b-kube-api-access-5fp48\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.239628 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.239612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv"
Nov 02 01:14:44.266895 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.266877    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:44.272026 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.272004    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:44.277990 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.277968    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:44.303840 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.303820    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:44.317678 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.317616    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68"
Nov 02 01:14:44.333119 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.333089    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:44.553436 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.553405    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p9c2l"]
Nov 02 01:14:44.556703 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:44.556676    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975d3545_32e4_4d97_a759_d4251bb28224.slice/crio-88c96a3e1e9db150e4c44314d2e37ddae56d98299dad5909a3baf46352e40b07 WatchSource:0}: Error finding container 88c96a3e1e9db150e4c44314d2e37ddae56d98299dad5909a3baf46352e40b07: Status 404 returned error can't find the container with id 88c96a3e1e9db150e4c44314d2e37ddae56d98299dad5909a3baf46352e40b07
Nov 02 01:14:44.579728 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.579706    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5jpp9"]
Nov 02 01:14:44.582758 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:44.582728    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50991da9_4137_467f_a695_0d0d4203321f.slice/crio-91c4902be7f9730a90a271ec2de33579d88bd130fc1a04dd3db08e721e53fc09 WatchSource:0}: Error finding container 91c4902be7f9730a90a271ec2de33579d88bd130fc1a04dd3db08e721e53fc09: Status 404 returned error can't find the container with id 91c4902be7f9730a90a271ec2de33579d88bd130fc1a04dd3db08e721e53fc09
Nov 02 01:14:44.622618 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.622556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:44.622618 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.622597    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:44.622808 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.622638    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:44.622808 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.622670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:44.622808 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.622717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:44.622964 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.622816    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:14:44.622964 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.622864    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls podName:91f2617a-9008-4f4a-ada4-8d5997064b52 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:45.622851516 +0000 UTC m=+32.543976834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2v57l" (UID: "91f2617a-9008-4f4a-ada4-8d5997064b52") : secret "samples-operator-tls" not found
Nov 02 01:14:44.623189 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.623174    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:44.623229 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.623190    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7978dd58c5-947pf: secret "image-registry-tls" not found
Nov 02 01:14:44.623229 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.623223    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls podName:d1b792a4-f821-4dec-9db4-36bfeceee2e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:45.623212899 +0000 UTC m=+32.544338215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls") pod "image-registry-7978dd58c5-947pf" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6") : secret "image-registry-tls" not found
Nov 02 01:14:44.623321 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.623267    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:45.623261052 +0000 UTC m=+32.544386366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : configmap references non-existent config key: service-ca.crt
Nov 02 01:14:44.623365 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.623330    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:14:44.623429 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.623367    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:45.623357295 +0000 UTC m=+32.544482615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : secret "router-metrics-certs-default" not found
Nov 02 01:14:44.623429 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.623415    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:44.623489 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.623444    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls podName:e57114c7-9426-4d83-870a-9d033cf335e4 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:45.623435248 +0000 UTC m=+32.544560563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gsw4b" (UID: "e57114c7-9426-4d83-870a-9d033cf335e4") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:44.623832 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.623810    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"]
Nov 02 01:14:44.625660 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.625637    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jhbp5"]
Nov 02 01:14:44.626261 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:44.626235    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414ef228_1302_4af5_8118_be23a4280494.slice/crio-a74ab7ffbf0290d9cb9063a70be636b69834e9a4bfcf86496dc3078d64891b8f WatchSource:0}: Error finding container a74ab7ffbf0290d9cb9063a70be636b69834e9a4bfcf86496dc3078d64891b8f: Status 404 returned error can't find the container with id a74ab7ffbf0290d9cb9063a70be636b69834e9a4bfcf86496dc3078d64891b8f
Nov 02 01:14:44.627919 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:44.627888    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d515403_65e0_4198_b78a_794ef628bc8f.slice/crio-6b773648e4feba6e5b8c5e28920c90ea97037e2d75222c3f8fe3d9a0244ebfe1 WatchSource:0}: Error finding container 6b773648e4feba6e5b8c5e28920c90ea97037e2d75222c3f8fe3d9a0244ebfe1: Status 404 returned error can't find the container with id 6b773648e4feba6e5b8c5e28920c90ea97037e2d75222c3f8fe3d9a0244ebfe1
Nov 02 01:14:44.667741 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.667720    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"]
Nov 02 01:14:44.670505 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.670485    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv"]
Nov 02 01:14:44.671509 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:44.671486    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc662357_7bcd_4efb_9857_37a197cb67bf.slice/crio-1805ead4e559c54faff6035c9255b9a5c708195aff930e9203598fb6d5168c30 WatchSource:0}: Error finding container 1805ead4e559c54faff6035c9255b9a5c708195aff930e9203598fb6d5168c30: Status 404 returned error can't find the container with id 1805ead4e559c54faff6035c9255b9a5c708195aff930e9203598fb6d5168c30
Nov 02 01:14:44.678531 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:44.678509    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7adf12_c300_4019_8425_6115d6ebca69.slice/crio-813d9b9dc23c67b5a55eefcefb4cf6d0ebb458b12d8d8240a7085d1e8e0a9057 WatchSource:0}: Error finding container 813d9b9dc23c67b5a55eefcefb4cf6d0ebb458b12d8d8240a7085d1e8e0a9057: Status 404 returned error can't find the container with id 813d9b9dc23c67b5a55eefcefb4cf6d0ebb458b12d8d8240a7085d1e8e0a9057
Nov 02 01:14:44.712232 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.712211    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"]
Nov 02 01:14:44.715421 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.715402    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bdwzr"]
Nov 02 01:14:44.725310 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.724576    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68"]
Nov 02 01:14:44.729530 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.729514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:44.730359 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.729698    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:44.730359 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.729711    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:44.730359 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.729731    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d8c5d8fd-g46xp: secret "image-registry-tls" not found
Nov 02 01:14:44.730359 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.729802    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls podName:fb996d25-e713-4ed5-a1db-871234d5a8cd nodeName:}" failed. No retries permitted until 2026-11-02 01:14:45.729776446 +0000 UTC m=+32.650901761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls") pod "image-registry-76d8c5d8fd-g46xp" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd") : secret "image-registry-tls" not found
Nov 02 01:14:44.730359 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.729826    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:44.730359 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.729867    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:14:44.730359 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.729929    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert podName:b2edcda9-40c1-4565-8843-1083f0f60b5a nodeName:}" failed. No retries permitted until 2026-11-02 01:14:45.729914617 +0000 UTC m=+32.651039918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9fbg8" (UID: "b2edcda9-40c1-4565-8843-1083f0f60b5a") : secret "networking-console-plugin-cert" not found
Nov 02 01:14:44.730359 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.729981    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:14:44.730359 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:44.730030    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls podName:1be28aae-f92c-4eb1-b063-f84b46967f3b nodeName:}" failed. No retries permitted until 2026-11-02 01:14:45.730021926 +0000 UTC m=+32.651147228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls") pod "dns-default-7zmf4" (UID: "1be28aae-f92c-4eb1-b063-f84b46967f3b") : secret "dns-default-metrics-tls" not found
Nov 02 01:14:44.748243 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.748220    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wlvr8"]
Nov 02 01:14:44.750545 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:44.750519    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51025d9a_739c_4934_9f5b_5bcbeb0d81ca.slice/crio-a924afcb201df0bdf5993b947246545049ff2a21b76e73195ea083925ca00270 WatchSource:0}: Error finding container a924afcb201df0bdf5993b947246545049ff2a21b76e73195ea083925ca00270: Status 404 returned error can't find the container with id a924afcb201df0bdf5993b947246545049ff2a21b76e73195ea083925ca00270
Nov 02 01:14:44.862841 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.862808    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wlvr8" event={"ID":"51025d9a-739c-4934-9f5b-5bcbeb0d81ca","Type":"ContainerStarted","Data":"a924afcb201df0bdf5993b947246545049ff2a21b76e73195ea083925ca00270"}
Nov 02 01:14:44.863921 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.863900    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68" event={"ID":"9fb03996-eac9-4517-9b19-283115a3ba00","Type":"ContainerStarted","Data":"6617b4113610a74824f141c8fbe72ce837d7d220b409231a44cc855c7b5a61e8"}
Nov 02 01:14:44.865035 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.865013    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr" event={"ID":"cc662357-7bcd-4efb-9857-37a197cb67bf","Type":"ContainerStarted","Data":"1805ead4e559c54faff6035c9255b9a5c708195aff930e9203598fb6d5168c30"}
Nov 02 01:14:44.866065 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.866045    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" event={"ID":"b395656d-d1a0-4cea-9b61-41788ab06732","Type":"ContainerStarted","Data":"436b0147f53478715945f04adbd00cd09d4a2255db9854bddae8023421bdec1d"}
Nov 02 01:14:44.867470 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.867455    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5jpp9" event={"ID":"50991da9-4137-467f-a695-0d0d4203321f","Type":"ContainerStarted","Data":"2b33750939bb9ac792d32d0e2a2631300c08f11414a80b793609381c1cedc8c2"}
Nov 02 01:14:44.867539 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.867485    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5jpp9" event={"ID":"50991da9-4137-467f-a695-0d0d4203321f","Type":"ContainerStarted","Data":"91c4902be7f9730a90a271ec2de33579d88bd130fc1a04dd3db08e721e53fc09"}
Nov 02 01:14:44.868690 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.868668    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bdwzr" event={"ID":"16b92078-b559-44f6-85b0-4f9d1da9ddcf","Type":"ContainerStarted","Data":"45ed23b73974914b493d80e358e44ab5a9678e6e26b23a8843f67a1c96a39f1c"}
Nov 02 01:14:44.869702 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.869683    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw" event={"ID":"414ef228-1302-4af5-8118-be23a4280494","Type":"ContainerStarted","Data":"a74ab7ffbf0290d9cb9063a70be636b69834e9a4bfcf86496dc3078d64891b8f"}
Nov 02 01:14:44.871027 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.871005    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-p9c2l" event={"ID":"975d3545-32e4-4d97-a759-d4251bb28224","Type":"ContainerStarted","Data":"83e76fddd342f9b025df8ef5422b7017b8e8e99cced4d92d443c5e13cc655454"}
Nov 02 01:14:44.871094 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.871043    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-p9c2l" event={"ID":"975d3545-32e4-4d97-a759-d4251bb28224","Type":"ContainerStarted","Data":"88c96a3e1e9db150e4c44314d2e37ddae56d98299dad5909a3baf46352e40b07"}
Nov 02 01:14:44.872090 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.872069    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv" event={"ID":"4b7adf12-c300-4019-8425-6115d6ebca69","Type":"ContainerStarted","Data":"813d9b9dc23c67b5a55eefcefb4cf6d0ebb458b12d8d8240a7085d1e8e0a9057"}
Nov 02 01:14:44.873524 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.873479    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jhbp5" event={"ID":"8d515403-65e0-4198-b78a-794ef628bc8f","Type":"ContainerStarted","Data":"f7d8bba82d02e9f8656f187fdbb9414e1868443cabc8e65fafbb97b078396f26"}
Nov 02 01:14:44.873524 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.873506    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jhbp5" event={"ID":"8d515403-65e0-4198-b78a-794ef628bc8f","Type":"ContainerStarted","Data":"6b773648e4feba6e5b8c5e28920c90ea97037e2d75222c3f8fe3d9a0244ebfe1"}
Nov 02 01:14:44.888570 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.888529    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5jpp9" podStartSLOduration=415.888518765 podStartE2EDuration="6m55.888518765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:44.887346163 +0000 UTC m=+31.808471490" watchObservedRunningTime="2026-11-02 01:14:44.888518765 +0000 UTC m=+31.809644086"
Nov 02 01:14:44.907230 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.907191    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jhbp5" podStartSLOduration=415.907180956 podStartE2EDuration="6m55.907180956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:44.906770486 +0000 UTC m=+31.827895813" watchObservedRunningTime="2026-11-02 01:14:44.907180956 +0000 UTC m=+31.828306279"
Nov 02 01:14:44.924991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:44.924959    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-p9c2l" podStartSLOduration=415.924947787 podStartE2EDuration="6m55.924947787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:44.924369149 +0000 UTC m=+31.845494475" watchObservedRunningTime="2026-11-02 01:14:44.924947787 +0000 UTC m=+31.846073111"
Nov 02 01:14:45.535378 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.535336    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:45.535809 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.535775    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:45.536102 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.536082    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:45.539412 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.539336    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:45.539742 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.539723    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wfrhp\""
Nov 02 01:14:45.539807 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.539793    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:14:45.539913 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.539897    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g7hgp\""
Nov 02 01:14:45.540026 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.540013    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:14:45.540119 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.540104    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7fmpb\""
Nov 02 01:14:45.639583 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.639555    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:45.640017 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.639600    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:45.640017 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.639643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:45.640017 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.639675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:45.640017 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.639716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:45.640017 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.639835    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:14:45.640017 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.639886    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls podName:91f2617a-9008-4f4a-ada4-8d5997064b52 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:47.639872704 +0000 UTC m=+34.560998019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2v57l" (UID: "91f2617a-9008-4f4a-ada4-8d5997064b52") : secret "samples-operator-tls" not found
Nov 02 01:14:45.640017 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.639941    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:45.640017 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.639949    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7978dd58c5-947pf: secret "image-registry-tls" not found
Nov 02 01:14:45.640017 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.639971    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls podName:d1b792a4-f821-4dec-9db4-36bfeceee2e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:47.639965036 +0000 UTC m=+34.561090350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls") pod "image-registry-7978dd58c5-947pf" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6") : secret "image-registry-tls" not found
Nov 02 01:14:45.640397 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.640034    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:47.640028366 +0000 UTC m=+34.561153681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : configmap references non-existent config key: service-ca.crt
Nov 02 01:14:45.640397 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.640074    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:14:45.640397 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.640097    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:47.640090481 +0000 UTC m=+34.561215797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : secret "router-metrics-certs-default" not found
Nov 02 01:14:45.640397 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.640133    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:45.640397 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.640156    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls podName:e57114c7-9426-4d83-870a-9d033cf335e4 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:47.640146782 +0000 UTC m=+34.561272099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gsw4b" (UID: "e57114c7-9426-4d83-870a-9d033cf335e4") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:45.715909 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.715881    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:45.741418 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.740127    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:45.741418 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.740231    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:45.741418 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.740473    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:45.741418 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.740488    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d8c5d8fd-g46xp: secret "image-registry-tls" not found
Nov 02 01:14:45.741418 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.740543    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls podName:fb996d25-e713-4ed5-a1db-871234d5a8cd nodeName:}" failed. No retries permitted until 2026-11-02 01:14:47.740526226 +0000 UTC m=+34.661651546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls") pod "image-registry-76d8c5d8fd-g46xp" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd") : secret "image-registry-tls" not found
Nov 02 01:14:45.741418 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.740854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:45.741418 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.741384    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:14:45.741418 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.741430    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls podName:1be28aae-f92c-4eb1-b063-f84b46967f3b nodeName:}" failed. No retries permitted until 2026-11-02 01:14:47.741414126 +0000 UTC m=+34.662539450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls") pod "dns-default-7zmf4" (UID: "1be28aae-f92c-4eb1-b063-f84b46967f3b") : secret "dns-default-metrics-tls" not found
Nov 02 01:14:45.741900 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.741444    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:14:45.741900 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:45.741478    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert podName:b2edcda9-40c1-4565-8843-1083f0f60b5a nodeName:}" failed. No retries permitted until 2026-11-02 01:14:47.74146828 +0000 UTC m=+34.662593597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9fbg8" (UID: "b2edcda9-40c1-4565-8843-1083f0f60b5a") : secret "networking-console-plugin-cert" not found
Nov 02 01:14:45.839956 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:45.839887    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:46.349274 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.347966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:46.349274 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.348110    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:46.349274 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:46.348914    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:14:46.349274 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:46.349027    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:18.349007372 +0000 UTC m=+65.270132697 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : secret "metrics-daemon-secret" not found
Nov 02 01:14:46.352703 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.352654    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:46.448745 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.448695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:46.454926 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.454897    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:46.474585 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.474553    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:46.496251 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.496019    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:46.806383 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.805636    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tkzsw"]
Nov 02 01:14:46.811509 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:46.811474    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cc7107_489a_4616_a157_e6321337b9be.slice/crio-6f0db25b3907c838c8a9c6d0f828a8e7c18609a76007e41ac07d929963d88755 WatchSource:0}: Error finding container 6f0db25b3907c838c8a9c6d0f828a8e7c18609a76007e41ac07d929963d88755: Status 404 returned error can't find the container with id 6f0db25b3907c838c8a9c6d0f828a8e7c18609a76007e41ac07d929963d88755
Nov 02 01:14:46.839749 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.839723    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nm87s"]
Nov 02 01:14:46.844620 ip-10-0-140-23 kubenswrapper[2569]: W1102 01:14:46.844593    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6da542_d39f_4a29_9849_08164ec387e0.slice/crio-ff3be16afe96139110677ab83296bc222bd6d027cbe69f123efe1ddd0250721b WatchSource:0}: Error finding container ff3be16afe96139110677ab83296bc222bd6d027cbe69f123efe1ddd0250721b: Status 404 returned error can't find the container with id ff3be16afe96139110677ab83296bc222bd6d027cbe69f123efe1ddd0250721b
Nov 02 01:14:46.897441 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.897407    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tkzsw" event={"ID":"12cc7107-489a-4616-a157-e6321337b9be","Type":"ContainerStarted","Data":"6f0db25b3907c838c8a9c6d0f828a8e7c18609a76007e41ac07d929963d88755"}
Nov 02 01:14:46.902252 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:46.902186    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nm87s" event={"ID":"7b6da542-d39f-4a29-9849-08164ec387e0","Type":"ContainerStarted","Data":"ff3be16afe96139110677ab83296bc222bd6d027cbe69f123efe1ddd0250721b"}
Nov 02 01:14:47.663996 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:47.663948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:47.664132 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:47.664018    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:47.664132 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:47.664080    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:47.664132 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:47.664125    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:47.664309 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:47.664180    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:47.665528 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.664429    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:14:47.665528 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.664503    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls podName:91f2617a-9008-4f4a-ada4-8d5997064b52 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.664481958 +0000 UTC m=+38.585607264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2v57l" (UID: "91f2617a-9008-4f4a-ada4-8d5997064b52") : secret "samples-operator-tls" not found
Nov 02 01:14:47.665528 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.665074    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:47.665528 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.665091    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7978dd58c5-947pf: secret "image-registry-tls" not found
Nov 02 01:14:47.665528 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.665142    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls podName:d1b792a4-f821-4dec-9db4-36bfeceee2e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.66512528 +0000 UTC m=+38.586250594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls") pod "image-registry-7978dd58c5-947pf" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6") : secret "image-registry-tls" not found
Nov 02 01:14:47.665528 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.665214    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.665202261 +0000 UTC m=+38.586327582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : configmap references non-existent config key: service-ca.crt
Nov 02 01:14:47.665528 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.665359    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:14:47.665528 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.665399    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.665386331 +0000 UTC m=+38.586511646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : secret "router-metrics-certs-default" not found
Nov 02 01:14:47.665528 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.665457    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:47.665528 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.665492    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls podName:e57114c7-9426-4d83-870a-9d033cf335e4 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.665480353 +0000 UTC m=+38.586605663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gsw4b" (UID: "e57114c7-9426-4d83-870a-9d033cf335e4") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:47.764794 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:47.764754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:47.764978 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:47.764867    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:47.764978 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:47.764945    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:47.765158 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.765126    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:14:47.765336 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.765187    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls podName:1be28aae-f92c-4eb1-b063-f84b46967f3b nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.765170061 +0000 UTC m=+38.686295377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls") pod "dns-default-7zmf4" (UID: "1be28aae-f92c-4eb1-b063-f84b46967f3b") : secret "dns-default-metrics-tls" not found
Nov 02 01:14:47.765655 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.765626    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:47.765655 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.765650    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d8c5d8fd-g46xp: secret "image-registry-tls" not found
Nov 02 01:14:47.765771 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.765696    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls podName:fb996d25-e713-4ed5-a1db-871234d5a8cd nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.765681366 +0000 UTC m=+38.686806686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls") pod "image-registry-76d8c5d8fd-g46xp" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd") : secret "image-registry-tls" not found
Nov 02 01:14:47.765771 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.765755    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:14:47.765845 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:47.765787    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert podName:b2edcda9-40c1-4565-8843-1083f0f60b5a nodeName:}" failed. No retries permitted until 2026-11-02 01:14:51.765775477 +0000 UTC m=+38.686900793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9fbg8" (UID: "b2edcda9-40c1-4565-8843-1083f0f60b5a") : secret "networking-console-plugin-cert" not found
Nov 02 01:14:47.910582 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:47.910494    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv" event={"ID":"4b7adf12-c300-4019-8425-6115d6ebca69","Type":"ContainerStarted","Data":"d0270c6ac38863c5fabc7c076f9bff76c0ec1615629c6d3f6db0f22c4b818e27"}
Nov 02 01:14:47.929016 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:47.928960    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv" podStartSLOduration=388.793853109 podStartE2EDuration="6m30.92894254s" totalImagesPullingTime="2.135089431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:47.928060812 +0000 UTC m=+34.849186137" watchObservedRunningTime="2026-11-02 01:14:47.92894254 +0000 UTC m=+34.850067905"
Nov 02 01:14:48.072280 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:48.072235    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:48.791917 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:48.791871    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:48.921563 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:48.921457    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:50.936671 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:50.935857    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr" event={"ID":"cc662357-7bcd-4efb-9857-37a197cb67bf","Type":"ContainerStarted","Data":"de01b9034a2e7e30c34f82c93752d0bb4741e83db51417c2152f2775f81adfba"}
Nov 02 01:14:50.938829 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:50.938810    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/0.log"
Nov 02 01:14:50.938965 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:50.938845    2569 generic.go:356] "Generic (PLEG): container finished" podID="b395656d-d1a0-4cea-9b61-41788ab06732" containerID="c4a0b44bcc2650b44f0033b365e6aa8ea2ed1f3c19f0d850d9b096cda30dfb00" exitCode=255
Nov 02 01:14:50.938965 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:50.938934    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" event={"ID":"b395656d-d1a0-4cea-9b61-41788ab06732","Type":"ContainerDied","Data":"c4a0b44bcc2650b44f0033b365e6aa8ea2ed1f3c19f0d850d9b096cda30dfb00"}
Nov 02 01:14:50.939416 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:50.939267    2569 scope.go:122] "RemoveContainer" containerID="c4a0b44bcc2650b44f0033b365e6aa8ea2ed1f3c19f0d850d9b096cda30dfb00"
Nov 02 01:14:50.941690 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:50.941317    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bdwzr" event={"ID":"16b92078-b559-44f6-85b0-4f9d1da9ddcf","Type":"ContainerStarted","Data":"c4fe19debf008433ac40c1c2e73686ae07e02f906a063c677031773da05224b3"}
Nov 02 01:14:50.943157 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:50.943134    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw" event={"ID":"414ef228-1302-4af5-8118-be23a4280494","Type":"ContainerStarted","Data":"58e5686e77211bdee714847c4d2a7f0b0041138bd6cc91e55964995d3291aa9e"}
Nov 02 01:14:50.957068 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:50.957021    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr" podStartSLOduration=387.993854679 podStartE2EDuration="6m33.957005272s" totalImagesPullingTime="5.963150593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:50.955720623 +0000 UTC m=+37.876845949" watchObservedRunningTime="2026-11-02 01:14:50.957005272 +0000 UTC m=+37.878130599"
Nov 02 01:14:51.016765 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.016453    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-bdwzr" podStartSLOduration=388.458368944 podStartE2EDuration="6m34.016434778s" totalImagesPullingTime="5.558065834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:50.998667866 +0000 UTC m=+37.919793205" watchObservedRunningTime="2026-11-02 01:14:51.016434778 +0000 UTC m=+37.937560105"
Nov 02 01:14:51.016942 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.016826    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw" podStartSLOduration=388.360851596 podStartE2EDuration="6m34.0168206s" totalImagesPullingTime="5.655969004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:51.015379066 +0000 UTC m=+37.936504390" watchObservedRunningTime="2026-11-02 01:14:51.0168206 +0000 UTC m=+37.937945924"
Nov 02 01:14:51.397991 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.395209    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m4wnq"]
Nov 02 01:14:51.416112 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.415970    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:51.429462 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.428348    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m4wnq"]
Nov 02 01:14:51.429462 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.428498    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.438607 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.436645    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 01:14:51.438607 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.436797    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hdjbk\""
Nov 02 01:14:51.473454 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.473426    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 01:14:51.508804 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.508768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.508804 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.508810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjwv\" (UniqueName: \"kubernetes.io/projected/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-api-access-9wjwv\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.509038 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.508890    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7363cd-3783-4f09-b5de-ce703edde1d5-crio-socket\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.509038 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.508916    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7363cd-3783-4f09-b5de-ce703edde1d5-data-volume\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.509038 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.508959    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.545481 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.545443    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.609891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjwv\" (UniqueName: \"kubernetes.io/projected/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-api-access-9wjwv\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.609996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7363cd-3783-4f09-b5de-ce703edde1d5-crio-socket\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.610039    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7363cd-3783-4f09-b5de-ce703edde1d5-data-volume\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.610079    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.610127    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.610212    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.610267    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:52.110248008 +0000 UTC m=+39.031373326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : configmap "kube-rbac-proxy" not found
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.610730    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7363cd-3783-4f09-b5de-ce703edde1d5-crio-socket\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.610991    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7363cd-3783-4f09-b5de-ce703edde1d5-data-volume\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.611059    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:51.611120 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.611097    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:52.11108706 +0000 UTC m=+39.032212375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:51.625734 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.625711    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjwv\" (UniqueName: \"kubernetes.io/projected/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-api-access-9wjwv\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:51.711376 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.711335    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:51.711600 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.711382    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:51.711600 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.711445    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:51.711600 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.711494    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:51.711600 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.711545    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:51.711800 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.711743    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:14:51.711854 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.711809    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls podName:91f2617a-9008-4f4a-ada4-8d5997064b52 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.711790059 +0000 UTC m=+46.632915377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2v57l" (UID: "91f2617a-9008-4f4a-ada4-8d5997064b52") : secret "samples-operator-tls" not found
Nov 02 01:14:51.712218 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.712189    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:51.712218 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.712208    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7978dd58c5-947pf: secret "image-registry-tls" not found
Nov 02 01:14:51.712415 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.712254    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls podName:d1b792a4-f821-4dec-9db4-36bfeceee2e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.71223956 +0000 UTC m=+46.633364877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls") pod "image-registry-7978dd58c5-947pf" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6") : secret "image-registry-tls" not found
Nov 02 01:14:51.712415 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.712337    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.712327364 +0000 UTC m=+46.633452679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : configmap references non-existent config key: service-ca.crt
Nov 02 01:14:51.712415 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.712396    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:14:51.712584 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.712428    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.712417646 +0000 UTC m=+46.633542962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : secret "router-metrics-certs-default" not found
Nov 02 01:14:51.712584 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.712495    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:51.712584 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.712531    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls podName:e57114c7-9426-4d83-870a-9d033cf335e4 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.712520506 +0000 UTC m=+46.633645822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gsw4b" (UID: "e57114c7-9426-4d83-870a-9d033cf335e4") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:51.712943 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.712922    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:51.812603 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.812569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:51.812752 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.812737    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:14:51.813160 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.813147    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls podName:1be28aae-f92c-4eb1-b063-f84b46967f3b nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.813111957 +0000 UTC m=+46.734237269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls") pod "dns-default-7zmf4" (UID: "1be28aae-f92c-4eb1-b063-f84b46967f3b") : secret "dns-default-metrics-tls" not found
Nov 02 01:14:51.813251 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.813212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:51.813321 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.813272    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:51.813469 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.813456    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:14:51.813512 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.813502    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert podName:b2edcda9-40c1-4565-8843-1083f0f60b5a nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.81349299 +0000 UTC m=+46.734618292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9fbg8" (UID: "b2edcda9-40c1-4565-8843-1083f0f60b5a") : secret "networking-console-plugin-cert" not found
Nov 02 01:14:51.813569 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.813562    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:51.813620 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.813572    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d8c5d8fd-g46xp: secret "image-registry-tls" not found
Nov 02 01:14:51.813620 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.813602    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls podName:fb996d25-e713-4ed5-a1db-871234d5a8cd nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.813591774 +0000 UTC m=+46.734717082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls") pod "image-registry-76d8c5d8fd-g46xp" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd") : secret "image-registry-tls" not found
Nov 02 01:14:51.850363 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.850330    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:51.949488 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.949351    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wlvr8" event={"ID":"51025d9a-739c-4934-9f5b-5bcbeb0d81ca","Type":"ContainerStarted","Data":"0ff7dffb41178cf0b5d506c65019be6be6b14441c64c9f5eb3ea15dd5b698738"}
Nov 02 01:14:51.951463 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.951440    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/1.log"
Nov 02 01:14:51.951970 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.951947    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/0.log"
Nov 02 01:14:51.952074 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.951987    2569 generic.go:356] "Generic (PLEG): container finished" podID="b395656d-d1a0-4cea-9b61-41788ab06732" containerID="3c8c978e03b521b7da25f87632d3c49708401650368c5a38f93953206af92671" exitCode=255
Nov 02 01:14:51.952424 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.952397    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" event={"ID":"b395656d-d1a0-4cea-9b61-41788ab06732","Type":"ContainerDied","Data":"3c8c978e03b521b7da25f87632d3c49708401650368c5a38f93953206af92671"}
Nov 02 01:14:51.952559 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.952457    2569 scope.go:122] "RemoveContainer" containerID="c4a0b44bcc2650b44f0033b365e6aa8ea2ed1f3c19f0d850d9b096cda30dfb00"
Nov 02 01:14:51.952874 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.952842    2569 scope.go:122] "RemoveContainer" containerID="3c8c978e03b521b7da25f87632d3c49708401650368c5a38f93953206af92671"
Nov 02 01:14:51.953057 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:51.953035    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cqpmz_openshift-console-operator(b395656d-d1a0-4cea-9b61-41788ab06732)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732"
Nov 02 01:14:51.971025 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:51.970974    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wlvr8" podStartSLOduration=2.365016362 podStartE2EDuration="8.970960729s" totalImagesPullingTime="6.605944367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:51.969483364 +0000 UTC m=+38.890608690" watchObservedRunningTime="2026-11-02 01:14:51.970960729 +0000 UTC m=+38.892086063"
Nov 02 01:14:52.120156 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:52.120108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:52.120379 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:52.120198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:52.120770 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:52.120469    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:52.120770 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:52.120491    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 01:14:52.120770 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:52.120560    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:53.120539392 +0000 UTC m=+40.041664710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:52.120770 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:52.120581    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:53.120570349 +0000 UTC m=+40.041695654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : configmap "kube-rbac-proxy" not found
Nov 02 01:14:52.723627 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:52.723582    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:52.957976 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:52.957938    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tkzsw" event={"ID":"12cc7107-489a-4616-a157-e6321337b9be","Type":"ContainerStarted","Data":"c5dd07251c3760c116af8af77587ae3af03d40ef016805ba5fb94d8a37e02509"}
Nov 02 01:14:52.958484 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:52.958034    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:52.959798 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:52.959772    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68" event={"ID":"9fb03996-eac9-4517-9b19-283115a3ba00","Type":"ContainerStarted","Data":"5059cd8b63548bf91450aadc4649248f172dcdf1c25885d90be9c099f875f65c"}
Nov 02 01:14:52.961480 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:52.961442    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/1.log"
Nov 02 01:14:52.961918 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:52.961896    2569 scope.go:122] "RemoveContainer" containerID="3c8c978e03b521b7da25f87632d3c49708401650368c5a38f93953206af92671"
Nov 02 01:14:52.962103 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:52.962081    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cqpmz_openshift-console-operator(b395656d-d1a0-4cea-9b61-41788ab06732)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732"
Nov 02 01:14:52.996881 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:52.996791    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tkzsw" podStartSLOduration=33.072441617 podStartE2EDuration="38.996772356s" totalImagesPullingTime="5.924330739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:52.976760245 +0000 UTC m=+39.897885605" watchObservedRunningTime="2026-11-02 01:14:52.996772356 +0000 UTC m=+39.917897681"
Nov 02 01:14:53.018499 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:53.018442    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68" podStartSLOduration=384.874996479 podStartE2EDuration="6m32.018425802s" totalImagesPullingTime="7.143429323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:53.017458796 +0000 UTC m=+39.938584122" watchObservedRunningTime="2026-11-02 01:14:53.018425802 +0000 UTC m=+39.939551128"
Nov 02 01:14:53.134596 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:53.134562    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:53.134789 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:53.134619    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:53.134789 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:53.134738    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:53.134789 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:53.134777    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 01:14:53.134941 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:53.134849    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:55.134828953 +0000 UTC m=+42.055954270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : configmap "kube-rbac-proxy" not found
Nov 02 01:14:53.134941 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:53.134868    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:55.13485883 +0000 UTC m=+42.055984148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:54.304064 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:54.304036    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:54.304456 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:54.304079    2569 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:54.304456 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:54.304399    2569 scope.go:122] "RemoveContainer" containerID="3c8c978e03b521b7da25f87632d3c49708401650368c5a38f93953206af92671"
Nov 02 01:14:54.304765 ip-10-0-140-23 kubenswrapper[2569]: E1102 01:14:54.304562    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cqpmz_openshift-console-operator(b395656d-d1a0-4cea-9b61-41788ab06732)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732"
Nov 02 01:14:54.600690 ip-10-0-140-23 kubenswrapper[2569]: I1102 01:14:54.600562    2569 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:14:54.600718 ip-10-0-140-23 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 01:14:54.618756 ip-10-0-140-23 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 01:14:54.618926 ip-10-0-140-23 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 01:14:54.618978 ip-10-0-140-23 systemd[1]: kubelet.service: Consumed 2.969s CPU time.
Nov 02 01:14:54.630909 ip-10-0-140-23 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 01:14:54.755871 ip-10-0-140-23 kubenswrapper[4851]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:14:54.755871 ip-10-0-140-23 kubenswrapper[4851]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 01:14:54.755871 ip-10-0-140-23 kubenswrapper[4851]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:14:54.755871 ip-10-0-140-23 kubenswrapper[4851]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:14:54.757995 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.757909    4851 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:54.757995 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.757994    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:54.757995 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.757998    4851 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758002    4851 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758006    4851 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758010    4851 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758015    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758018    4851 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758021    4851 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758024    4851 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758027    4851 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758029    4851 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758032    4851 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758034    4851 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758037    4851 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758039    4851 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758042    4851 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758045    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758047    4851 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758050    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758052    4851 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:54.758088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758055    4851 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758058    4851 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758061    4851 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758063    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758066    4851 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758071    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758074    4851 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758077    4851 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758079    4851 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758082    4851 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758085    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758088    4851 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758091    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758094    4851 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758097    4851 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758100    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758103    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758106    4851 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758108    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:54.758593 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758111    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758113    4851 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758116    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758119    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758121    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758124    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758126    4851 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758129    4851 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758132    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758134    4851 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758137    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758139    4851 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758143    4851 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758146    4851 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758149    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758152    4851 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758154    4851 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758157    4851 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758160    4851 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758163    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:54.759080 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758166    4851 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758169    4851 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758171    4851 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758174    4851 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758177    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758180    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758182    4851 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758185    4851 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758188    4851 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758190    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758193    4851 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758196    4851 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758199    4851 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758202    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758204    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758207    4851 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758209    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758212    4851 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758215    4851 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:54.759607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758218    4851 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758220    4851 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758222    4851 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758225    4851 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758228    4851 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758231    4851 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758233    4851 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758236    4851 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758239    4851 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758241    4851 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758244    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758247    4851 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758249    4851 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758251    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758254    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758256    4851 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758260    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758265    4851 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758269    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:54.760094 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758272    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758275    4851 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758278    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758280    4851 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758283    4851 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758285    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758304    4851 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.758307    4851 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759055    4851 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759062    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759066    4851 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759071    4851 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759076    4851 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759080    4851 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759083    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759086    4851 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759089    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759092    4851 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759094    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:54.760604 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759097    4851 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759100    4851 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759103    4851 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759106    4851 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759108    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759111    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759113    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759116    4851 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759119    4851 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759122    4851 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759125    4851 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759128    4851 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759131    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759134    4851 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759136    4851 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759139    4851 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759141    4851 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759144    4851 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759147    4851 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759150    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:54.761114 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759152    4851 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759155    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759157    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759160    4851 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759163    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759165    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759168    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759171    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759174    4851 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759176    4851 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759179    4851 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759182    4851 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759184    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759187    4851 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759189    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759192    4851 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759195    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759197    4851 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759200    4851 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759202    4851 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:54.761635 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759204    4851 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759207    4851 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759210    4851 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759213    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759216    4851 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759219    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759222    4851 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759225    4851 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759227    4851 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759230    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759232    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759235    4851 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759237    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759240    4851 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759242    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759245    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759247    4851 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759250    4851 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759252    4851 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:54.762161 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759255    4851 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759258    4851 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759260    4851 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759263    4851 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759265    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759268    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759270    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759273    4851 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759275    4851 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759278    4851 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759280    4851 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759285    4851 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759304    4851 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759306    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759309    4851 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759312    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759315    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759318    4851 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759321    4851 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:54.762657 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759325    4851 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759328    4851 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759331    4851 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759334    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759336    4851 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759339    4851 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759341    4851 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759343    4851 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759346    4851 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759348    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759351    4851 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759354    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759357    4851 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759359    4851 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759362    4851 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759365    4851 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759367    4851 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759838    4851 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759843    4851 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759846    4851 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:54.763142 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759849    4851 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759852    4851 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759854    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759857    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759862    4851 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759865    4851 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759867    4851 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759871    4851 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759873    4851 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759876    4851 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759879    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759881    4851 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759884    4851 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759886    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759889    4851 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759892    4851 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759894    4851 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759897    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759900    4851 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759902    4851 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:54.763695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759905    4851 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759907    4851 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759910    4851 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759912    4851 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759915    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759918    4851 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759920    4851 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759924    4851 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759927    4851 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759930    4851 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759932    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759935    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759937    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759939    4851 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759942    4851 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759944    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759949    4851 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759951    4851 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759954    4851 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:54.764199 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759957    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759959    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759962    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759965    4851 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759967    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759970    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759972    4851 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759975    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759978    4851 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759980    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759983    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759985    4851 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759988    4851 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759990    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759993    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759995    4851 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.759998    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760000    4851 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760003    4851 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760005    4851 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:54.764689 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760008    4851 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760014    4851 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760017    4851 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760019    4851 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760022    4851 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760024    4851 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760027    4851 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760042    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760047    4851 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760052    4851 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760055    4851 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760058    4851 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760061    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760064    4851 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760067    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760069    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760072    4851 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760075    4851 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760078    4851 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:54.765188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760080    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760083    4851 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760085    4851 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760088    4851 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760090    4851 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760093    4851 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760095    4851 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760098    4851 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760101    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760104    4851 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760106    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760109    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760111    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760114    4851 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760116    4851 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760119    4851 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760122    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760126    4851 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760130    4851 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760133    4851 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:54.765694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760136    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760138    4851 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760142    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760145    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760148    4851 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760239    4851 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760250    4851 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760255    4851 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760260    4851 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760264    4851 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760267    4851 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760271    4851 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760275    4851 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760278    4851 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760282    4851 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760285    4851 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760304    4851 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760308    4851 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760311    4851 flags.go:64] FLAG: --cgroup-root=""
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760313    4851 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760317    4851 flags.go:64] FLAG: --client-ca-file=""
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760320    4851 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760323    4851 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 01:14:54.766212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760327    4851 flags.go:64] FLAG: --cluster-domain=""
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760330    4851 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760333    4851 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760336    4851 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760339    4851 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760344    4851 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760347    4851 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760350    4851 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760353    4851 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760357    4851 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760360    4851 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760363    4851 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760367    4851 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760369    4851 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760374    4851 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760377    4851 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760379    4851 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760382    4851 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760385    4851 flags.go:64] FLAG: --enable-server="true"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760388    4851 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760392    4851 flags.go:64] FLAG: --event-burst="100"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760395    4851 flags.go:64] FLAG: --event-qps="50"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760398    4851 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760401    4851 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 01:14:54.766807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760405    4851 flags.go:64] FLAG: --eviction-hard=""
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760410    4851 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760413    4851 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760416    4851 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760419    4851 flags.go:64] FLAG: --eviction-soft=""
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760421    4851 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760424    4851 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760427    4851 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760430    4851 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760433    4851 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760436    4851 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760439    4851 flags.go:64] FLAG: --feature-gates=""
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760442    4851 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760445    4851 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760449    4851 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760452    4851 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760455    4851 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760459    4851 flags.go:64] FLAG: --help="false"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760461    4851 flags.go:64] FLAG: --hostname-override="ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760466    4851 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760469    4851 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760472    4851 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760475    4851 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 01:14:54.767437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760478    4851 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760481    4851 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760484    4851 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760487    4851 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760490    4851 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760493    4851 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760496    4851 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760498    4851 flags.go:64] FLAG: --kube-reserved=""
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760501    4851 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760504    4851 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760508    4851 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760511    4851 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760514    4851 flags.go:64] FLAG: --lock-file=""
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760517    4851 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760519    4851 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760523    4851 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760528    4851 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760531    4851 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760534    4851 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760536    4851 flags.go:64] FLAG: --logging-format="text"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760539    4851 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760542    4851 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760545    4851 flags.go:64] FLAG: --manifest-url=""
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760548    4851 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760553    4851 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 01:14:54.768019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760556    4851 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760559    4851 flags.go:64] FLAG: --max-pods="110"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760563    4851 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760567    4851 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760570    4851 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760573    4851 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760576    4851 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760579    4851 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760582    4851 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760589    4851 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760592    4851 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760595    4851 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760597    4851 flags.go:64] FLAG: --pod-cidr=""
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760600    4851 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760603    4851 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760606    4851 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760609    4851 flags.go:64] FLAG: --port="10250"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760613    4851 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760617    4851 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-058a6e3a4adaf3bbf"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760620    4851 flags.go:64] FLAG: --qos-reserved=""
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760623    4851 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760625    4851 flags.go:64] FLAG: --register-node="true"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760629    4851 flags.go:64] FLAG: --register-with-taints=""
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760632    4851 flags.go:64] FLAG: --registry-burst="10"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760635    4851 flags.go:64] FLAG: --registry-qps="5"
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760638    4851 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 01:14:54.768671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760641    4851 flags.go:64] FLAG: --reserved-memory=""
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760645    4851 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760648    4851 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760651    4851 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760654    4851 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760656    4851 flags.go:64] FLAG: --runonce="false"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760660    4851 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760663    4851 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760666    4851 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760670    4851 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760672    4851 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760675    4851 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760678    4851 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760681    4851 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760684    4851 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760687    4851 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760689    4851 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760692    4851 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760695    4851 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760698    4851 flags.go:64] FLAG: --system-cgroups=""
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760700    4851 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760705    4851 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760708    4851 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760712    4851 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760716    4851 flags.go:64] FLAG: --tls-min-version=""
Nov 02 01:14:54.769700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760719    4851 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760721    4851 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760724    4851 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760727    4851 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760730    4851 flags.go:64] FLAG: --v="2"
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760734    4851 flags.go:64] FLAG: --version="false"
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760738    4851 flags.go:64] FLAG: --vmodule=""
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760742    4851 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.760745    4851 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760865    4851 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760868    4851 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760871    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760874    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760876    4851 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760879    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760882    4851 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760884    4851 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760887    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760890    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760892    4851 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760895    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760897    4851 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:54.770515 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760900    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760903    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760905    4851 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760908    4851 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760910    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760913    4851 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760915    4851 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760918    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760921    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760924    4851 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760927    4851 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760929    4851 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760932    4851 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760934    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760937    4851 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760939    4851 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760942    4851 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760944    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760947    4851 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760949    4851 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:54.771097 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760952    4851 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760954    4851 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760957    4851 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760959    4851 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760962    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760965    4851 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760967    4851 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760970    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760972    4851 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760974    4851 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760977    4851 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760980    4851 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760982    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760985    4851 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760987    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760990    4851 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760992    4851 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760995    4851 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.760999    4851 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:54.771664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761002    4851 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761007    4851 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761010    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761013    4851 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761015    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761018    4851 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761020    4851 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761023    4851 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761025    4851 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761028    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761030    4851 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761033    4851 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761035    4851 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761038    4851 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761040    4851 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761042    4851 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761045    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761047    4851 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761050    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761059    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:54.772215 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761061    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761064    4851 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761066    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761069    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761073    4851 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761076    4851 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761078    4851 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761081    4851 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761084    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761086    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761089    4851 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761092    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761094    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761098    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761101    4851 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761104    4851 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761106    4851 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761109    4851 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761112    4851 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:54.772740 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761114    4851 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761117    4851 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761120    4851 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761122    4851 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761125    4851 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761129    4851 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761132    4851 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761135    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761138    4851 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761140    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761143    4851 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761146    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761148    4851 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761152    4851 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761155    4851 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.761179    4851 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:14:54.773335 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761214    4851 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761219    4851 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761223    4851 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761226    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761229    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761232    4851 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761235    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761237    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761240    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761243    4851 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761247    4851 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761249    4851 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761252    4851 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761255    4851 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761257    4851 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761260    4851 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761262    4851 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761265    4851 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761268    4851 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:54.773946 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761271    4851 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761275    4851 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761277    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761280    4851 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761282    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761285    4851 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761300    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761303    4851 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761306    4851 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761308    4851 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761311    4851 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761314    4851 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761317    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761320    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761322    4851 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761326    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761329    4851 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761331    4851 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761334    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761337    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:54.774559 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761340    4851 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761342    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761345    4851 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761349    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761351    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761354    4851 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761357    4851 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761359    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761362    4851 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761365    4851 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761367    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761370    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761373    4851 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761375    4851 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761378    4851 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761380    4851 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761383    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761386    4851 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761388    4851 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:54.775064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761391    4851 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761394    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761397    4851 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761400    4851 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761402    4851 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761405    4851 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761408    4851 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761410    4851 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761416    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761419    4851 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761421    4851 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761424    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761426    4851 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761429    4851 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761432    4851 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761434    4851 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761438    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761440    4851 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761443    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761446    4851 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:54.775678 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761448    4851 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761451    4851 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761454    4851 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761458    4851 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761462    4851 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761465    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761467    4851 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761470    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761472    4851 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761475    4851 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761477    4851 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761480    4851 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761482    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761486    4851 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761488    4851 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761491    4851 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761493    4851 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761496    4851 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761499    4851 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:54.776309 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761502    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761506    4851 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761509    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761511    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761514    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761516    4851 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761519    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761522    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.761525    4851 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.765240    4851 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.765256    4851 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.765275    4851 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.765279    4851 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765350    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765355    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765359    4851 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765362    4851 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765364    4851 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765367    4851 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:54.776880 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765371    4851 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765374    4851 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765378    4851 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765382    4851 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765385    4851 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765388    4851 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765391    4851 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765393    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765396    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765398    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765401    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765404    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765407    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765410    4851 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765412    4851 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765415    4851 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765417    4851 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765420    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765423    4851 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:54.777541 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765425    4851 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765428    4851 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765431    4851 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765433    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765437    4851 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765440    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765442    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765445    4851 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765448    4851 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765451    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765453    4851 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765456    4851 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765459    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765461    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765464    4851 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765466    4851 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765469    4851 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765471    4851 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765474    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765476    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:54.778180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765478    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765481    4851 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765483    4851 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765486    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765489    4851 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765491    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765494    4851 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765497    4851 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765500    4851 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765503    4851 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765505    4851 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765508    4851 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765511    4851 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765513    4851 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765516    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765519    4851 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765522    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765525    4851 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765527    4851 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765530    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:54.778824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765532    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765535    4851 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765537    4851 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765540    4851 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765542    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765545    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765547    4851 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765550    4851 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765552    4851 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765555    4851 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765558    4851 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765560    4851 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765564    4851 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765567    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765570    4851 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765574    4851 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765577    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765580    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765582    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:54.779323 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765585    4851 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765587    4851 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765590    4851 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765592    4851 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765595    4851 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765597    4851 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765600    4851 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765603    4851 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765606    4851 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765608    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765610    4851 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765613    4851 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765616    4851 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765618    4851 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765621    4851 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765623    4851 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765626    4851 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765628    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765631    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:54.779783 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765633    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765636    4851 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765638    4851 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.765666    4851 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765806    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765810    4851 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765813    4851 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765815    4851 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765818    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765820    4851 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765823    4851 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765826    4851 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765828    4851 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765831    4851 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765833    4851 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765836    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:54.780247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765838    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765841    4851 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765843    4851 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765846    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765848    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765851    4851 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765854    4851 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765856    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765859    4851 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765862    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765865    4851 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765867    4851 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765870    4851 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765872    4851 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765874    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765877    4851 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765879    4851 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765882    4851 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765884    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765887    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:54.780655 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765889    4851 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765892    4851 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765895    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765897    4851 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765900    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765902    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765906    4851 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765911    4851 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765914    4851 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765917    4851 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765919    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765922    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765924    4851 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765927    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765929    4851 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765932    4851 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765934    4851 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765937    4851 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765939    4851 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:54.781126 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765942    4851 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765944    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765947    4851 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765950    4851 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765952    4851 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765955    4851 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765957    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765960    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765964    4851 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765968    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765971    4851 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765974    4851 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765976    4851 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765980    4851 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765983    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765986    4851 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765988    4851 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765991    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765993    4851 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:54.781606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765996    4851 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.765999    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766001    4851 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766004    4851 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766006    4851 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766009    4851 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766012    4851 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766015    4851 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766018    4851 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766020    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766022    4851 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766025    4851 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766027    4851 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766030    4851 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766032    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766034    4851 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766037    4851 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766040    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766042    4851 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766045    4851 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:54.782071 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766047    4851 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766050    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766052    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766055    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766057    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766060    4851 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766062    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766064    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766067    4851 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766072    4851 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766075    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766077    4851 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766080    4851 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766082    4851 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766085    4851 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:54.782755 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766088    4851 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.766110    4851 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766142    4851 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766146    4851 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766149    4851 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766153    4851 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766156    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766159    4851 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766161    4851 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766164    4851 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766166    4851 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766169    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766171    4851 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766174    4851 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766177    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766180    4851 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:14:54.783156 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766182    4851 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766185    4851 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766188    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766190    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766193    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766195    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766198    4851 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766200    4851 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766202    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766205    4851 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766208    4851 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766211    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766214    4851 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766217    4851 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766219    4851 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766222    4851 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766225    4851 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766228    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766230    4851 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:14:54.783914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766233    4851 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766235    4851 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766238    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766240    4851 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766243    4851 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766245    4851 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766247    4851 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766250    4851 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766253    4851 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766255    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766258    4851 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766260    4851 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766263    4851 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766266    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766269    4851 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766271    4851 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766274    4851 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766276    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766279    4851 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766281    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:14:54.784692 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766284    4851 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766301    4851 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766304    4851 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766306    4851 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766310    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766313    4851 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766315    4851 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766318    4851 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766320    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766323    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766325    4851 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766328    4851 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766331    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766333    4851 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766336    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766339    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766342    4851 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766346    4851 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766348    4851 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:14:54.785506 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766351    4851 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766354    4851 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766357    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766360    4851 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766362    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766365    4851 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766367    4851 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766370    4851 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766373    4851 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766376    4851 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766378    4851 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766381    4851 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766383    4851 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766385    4851 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766388    4851 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766390    4851 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766393    4851 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766397    4851 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766401    4851 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766405    4851 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:14:54.786060 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766408    4851 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766411    4851 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766414    4851 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766416    4851 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766419    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766422    4851 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766424    4851 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766427    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766429    4851 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766432    4851 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766434    4851 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766437    4851 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766439    4851 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:54.766441    4851 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.766489    4851 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.767406    4851 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.767450    4851 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.767631    4851 server.go:1014] "Starting client certificate rotation"
Nov 02 01:14:54.786572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.767788    4851 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:14:54.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.767834    4851 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:09:14 +0000 UTC" deadline="2028-05-09 16:57:02.14925108 +0000 UTC"
Nov 02 01:14:54.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.767880    4851 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13311h42m7.381373854s"
Nov 02 01:14:54.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.768140    4851 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:14:54.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.770315    4851 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:14:54.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.771466    4851 log.go:25] "Validated CRI v1 runtime API"
Nov 02 01:14:54.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.772987    4851 log.go:25] "Validated CRI v1 image API"
Nov 02 01:14:54.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.773771    4851 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 01:14:54.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.775660    4851 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 dab46697-ff91-48bc-a185-0a71b8409249:/dev/nvme0n1p4 f572e20b-9980-404d-947a-7909eed376a8:/dev/nvme0n1p3]
Nov 02 01:14:54.787321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.775678    4851 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10fb972b8d11e03e3c10123b38da659288991e0786c0a985e169cc2a79c65c24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10fb972b8d11e03e3c10123b38da659288991e0786c0a985e169cc2a79c65c24/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1805ead4e559c54faff6035c9255b9a5c708195aff930e9203598fb6d5168c30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1805ead4e559c54faff6035c9255b9a5c708195aff930e9203598fb6d5168c30/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c2fae246ea9e2354173d0be93469c0dce2ae78e79e3f6f37f8d8ff5272bbdbb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c2fae246ea9e2354173d0be93469c0dce2ae78e79e3f6f37f8d8ff5272bbdbb/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/436b0147f53478715945f04adbd00cd09d4a2255db9854bddae8023421bdec1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/436b0147f53478715945f04adbd00cd09d4a2255db9854bddae8023421bdec1d/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45ed23b73974914b493d80e358e44ab5a9678e6e26b23a8843f67a1c96a39f1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45ed23b73974914b493d80e358e44ab5a9678e6e26b23a8843f67a1c96a39f1c/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6617b4113610a74824f141c8fbe72ce837d7d220b409231a44cc855c7b5a61e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6617b4113610a74824f141c8fbe72ce837d7d220b409231a44cc855c7b5a61e8/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b773648e4feba6e5b8c5e28920c90ea97037e2d75222c3f8fe3d9a0244ebfe1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b773648e4feba6e5b8c5e28920c90ea97037e2d75222c3f8fe3d9a0244ebfe1/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f0db25b3907c838c8a9c6d0f828a8e7c18609a76007e41ac07d929963d88755/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f0db25b3907c838c8a9c6d0f828a8e7c18609a76007e41ac07d929963d88755/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79b9e8663e48dfa9bec608ecd319d8aef0f9a899aa0b80e0866881a57373953b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79b9e8663e48dfa9bec608ecd319d8aef0f9a899aa0b80e0866881a57373953b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fb18e18bdef725db03e0cdccb78cdbb10e2a4be6a9bf50facafb13b85813dee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fb18e18bdef725db03e0cdccb78cdbb10e2a4be6a9bf50facafb13b85813dee/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/813d9b9dc23c67b5a55eefcefb4cf6d0ebb458b12d8d8240a7085d1e8e0a9057/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/813d9b9dc23c67b5a55eefcefb4cf6d0ebb458b12d8d8240a7085d1e8e0a9057/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88c96a3e1e9db150e4c44314d2e37ddae56d98299dad5909a3baf46352e40b07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88c96a3e1e9db150e4c44314d2e37ddae56d98299dad5909a3baf46352e40b07/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a29820c4984e659393d0f60e28378c39745227abfecbb12e5df400a06af2bd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a29820c4984e659393d0f60e28378c39745227abfecbb12e5df400a06af2bd3/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91c4902be7f9730a90a271ec2de33579d88bd130fc1a04dd3db08e721e53fc09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91c4902be7f9730a90a271ec2de33579d88bd130fc1a04dd3db08e721e53fc09/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94b89615a53092de60a9fb64c3500c8748b614bcee93733bc4eee294846f1b4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94b89615a53092de60a9fb64c3500c8748b614bcee93733bc4eee294846f1b4b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/959d6c6c6831aca9731a1a9a29a3df47943b5c55e37372d60107742c14547efc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/959d6c6c6831aca9731a1a9a29a3df47943b5c55e37372d60107742c14547efc/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9da9a4bca090f45342cbeb031e9fd07da4ece5a9e0698a480b520c77cec098f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9da9a4bca090f45342cbeb031e9fd07da4ece5a9e0698a480b520c77cec098f3/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a74ab7ffbf0290d9cb9063a70be636b69834e9a4bfcf86496dc3078d64891b8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a74ab7ffbf0290d9cb9063a70be636b69834e9a4bfcf86496dc3078d64891b8f/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a924afcb201df0bdf5993b947246545049ff2a21b76e73195ea083925ca00270/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a924afcb201df0bdf5993b947246545049ff2a21b76e73195ea083925ca00270/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cac6b082d49b82a25c0009cc24f65f321bf19a23b063a79ac924cc87450c3e94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cac6b082d49b82a25c0009cc24f65f321bf19a23b063a79ac924cc87450c3e94/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0dcefa6154dda1cd27789613274e478e25dbe925f881ddf2788705d8ffe2b14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0dcefa6154dda1cd27789613274e478e25dbe925f881ddf2788705d8ffe2b14/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff3be16afe96139110677ab83296bc222bd6d027cbe69f123efe1ddd0250721b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff3be16afe96139110677ab83296bc222bd6d027cbe69f123efe1ddd0250721b/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ccf76ab-a4c9-4087-8ada-59b4c19218f8/volumes/kubernetes.io~projected/kube-api-access-n5whv:{mountpoint:/var/lib/kubelet/pods/0ccf76ab-a4c9-4087-8ada-59b4c19218f8/volumes/kubernetes.io~projected/kube-api-access-n5whv major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12cc7107-489a-4616-a157-e6321337b9be/volumes/kubernetes.io~projected/kube-api-access-6bq9l:{mountpoint:/var/lib/kubelet/pods/12cc7107-489a-4616-a157-e6321337b9be/volumes/kubernetes.io~projected/kube-api-access-6bq9l major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16b92078-b559-44f6-85b0-4f9d1da9ddcf/volumes/kubernetes.io~projected/kube-api-access-sg6v7:{mountpoint:/var/lib/kubelet/pods/16b92078-b559-44f6-85b0-4f9d1da9ddcf/volumes/kubernetes.io~projected/kube-api-access-sg6v7 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16b92078-b559-44f6-85b0-4f9d1da9ddcf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/16b92078-b559-44f6-85b0-4f9d1da9ddcf/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1be28aae-f92c-4eb1-b063-f84b46967f3b/volumes/kubernetes.io~projected/kube-api-access-5fp48:{mountpoint:/var/lib/kubelet/pods/1be28aae-f92c-4eb1-b063-f84b46967f3b/volumes/kubernetes.io~projected/kube-api-access-5fp48 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ad10cee-ccd8-424d-8e5b-8eed99b3881e/volumes/kubernetes.io~projected/kube-api-access-ztlqj:{mountpoint:/var/lib/kubelet/pods/3ad10cee-ccd8-424d-8e5b-8eed99b3881e/volumes/kubernetes.io~projected/kube-api-access-ztlqj major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/414ef228-1302-4af5-8118-be23a4280494/volumes/kubernetes.io~projected/kube-api-access-w5krs:{mountpoint:/var/lib/kubelet/pods/414ef228-1302-4af5-8118-be23a4280494/volumes/kubernetes.io~projected/kube-api-access-w5krs major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/414ef228-1302-4af5-8118-be23a4280494/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/414ef228-1302-4af5-8118-be23a4280494/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b7adf12-c300-4019-8425-6115d6ebca69/volumes/kubernetes.io~projected/kube-api-access-2pqb4:{mountpoint:/var/lib/kubelet/pods/4b7adf12-c300-4019-8425-6115d6ebca69/volumes/kubernetes.io~projected/kube-api-access-2pqb4 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50991da9-4137-467f-a695-0d0d4203321f/volumes/kubernetes.io~projected/kube-api-access-j7wxp:{mountpoint:/var/lib/kubelet/pods/50991da9-4137-467f-a695-0d0d4203321f/volumes/kubernetes.io~projected/kube-api-access-j7wxp major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51025d9a-739c-4934-9f5b-5bcbeb0d81ca/volumes/kubernetes.io~projected/kube-api-access-27wxr:{mountpoint:/var/lib/kubelet/pods/51025d9a-739c-4934-9f5b-5bcbeb0d81ca/volumes/kubernetes.io~projected/kube-api-access-27wxr major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51025d9a-739c-4934-9f5b-5bcbeb0d81ca/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/51025d9a-739c-4934-9f5b-5bcbeb0d81ca/volumes/kubernetes.io~secret/cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d45205b-f684-4607-b594-0352c721a4f7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5d45205b-f684-4607-b594-0352c721a4f7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d45205b-f684-4607-b594-0352c721a4f7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5d45205b-f684-4607-b594-0352c721a4f7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d45205b-f684-4607-b594-0352c721a4f7/volumes/kubernetes.io~projected/kube-api-access-4l4h9:{mountpoint:/var/lib/kubelet/pods/5d45205b-f684-4607-b594-0352c721a4f7/volumes/kubernetes.io~projected/kube-api-access-4l4h9 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83/volumes/kubernetes.io~projected/kube-api-access-t4ssf:{mountpoint:/var/lib/kubelet/pods/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83/volumes/kubernetes.io~projected/kube-api-access-t4ssf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68a83bf9-59f7-42a3-aeb0-efc8691a6644/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/68a83bf9-59f7-42a3-aeb0-efc8691a6644/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68a83bf9-59f7-42a3-aeb0-efc8691a6644/volumes/kubernetes.io~projected/kube-api-access-2cphg:{mountpoint:/var/lib/kubelet/pods/68a83bf9-59f7-42a3-aeb0-efc8691a6644/volumes/kubernetes.io~projected/kube-api-access-2cphg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68a83bf9-59f7-42a3-aeb0-efc8691a6644/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/68a83bf9-59f7-42a3-aeb0-efc8691a6644/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b6da542-d39f-4a29-9849-08164ec387e0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7b6da542-d39f-4a29-9849-08164ec387e0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d515403-65e0-4198-b78a-794ef628bc8f/volumes/kubernetes.io~projected/kube-api-access-jkrqr:{mountpoint:/var/lib/kubelet/pods/8d515403-65e0-4198-b78a-794ef628bc8f/volumes/kubernetes.io~projected/kube-api-access-jkrqr major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91f2617a-9008-4f4a-ada4-8d5997064b52/volumes/kubernetes.io~projected/kube-api-access-sbmk5:{mountpoint:/var/lib/kubelet/pods/91f2617a-9008-4f4a-ada4-8d5997064b52/volumes/kubernetes.io~projected/kube-api-access-sbmk5 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96b08fc5-cc99-4228-9a22-1e57c7fed18b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/96b08fc5-cc99-4228-9a22-1e57c7fed18b/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/975d3545-32e4-4d97-a759-d4251bb28224/volumes/kubernetes.io~projected/kube-api-access-bd26r:{mountpoint:/var/lib/kubelet/pods/975d3545-32e4-4d97-a759-d4251bb28224/volumes/kubernetes.io~projected/kube-api-access-bd26r major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb03996-eac9-4517-9b19-283115a3ba00/volumes/kubernetes.io~projected/kube-api-access-qld9n:{mountpoint:/var/lib/kubelet/pods/9fb03996-eac9-4517-9b19-283115a3ba00/volumes/kubernetes.io~projected/kube-api-access-qld9n major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab7363cd-3783-4f09-b5de-ce703edde1d5/volumes/kubernetes.io~projected/kube-api-access-9wjwv:{mountpoint:/var/lib/kubelet/pods/ab7363cd-3783-4f09-b5de-ce703edde1d5/volumes/kubernetes.io~projected/kube-api-access-9wjwv major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b395656d-d1a0-4cea-9b61-41788ab06732/volumes/kubernetes.io~projected/kube-api-access-pkffz:{mountpoint:/var/lib/kubelet/pods/b395656d-d1a0-4cea-9b61-41788ab06732/volumes/kubernetes.io~projected/kube-api-access-pkffz major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b395656d-d1a0-4cea-9b61-41788ab06732/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b395656d-d1a0-4cea-9b61-41788ab06732/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f/volumes/kubernetes.io~projected/kube-api-access-rwlm2:{mountpoint:/var/lib/kubelet/pods/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f/volumes/kubernetes.io~projected/kube-api-access-rwlm2 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4afb19d-ffe5-4011-81a0-5761db34bd75/volumes/kubernetes.io~projected/kube-api-access-cn89l:{mountpoint:/var/lib/kubelet/pods/c4afb19d-ffe5-4011-81a0-5761db34bd75/volumes/kubernetes.io~projected/kube-api-access-cn89l major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc662357-7bcd-4efb-9857-37a197cb67bf/volumes/kubernetes.io~projected/kube-api-access-87d7t:{mountpoint:/var/lib/kubelet/pods/cc662357-7bcd-4efb-9857-37a197cb67bf/volumes/kubernetes.io~projected/kube-api-access-87d7t major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc662357-7bcd-4efb-9857-37a197cb67bf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cc662357-7bcd-4efb-9857-37a197cb67bf/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce7ea8b0-06fd-457e-804b-da8c4f73edfc/volumes/kubernetes.io~projected/kube-api-access-dnfhv:{mountpoint:/var/lib/kubelet/pods/ce7ea8b0-06fd-457e-804b-da8c4f73edfc/volumes/kubernetes.io~projected/kube-api-access-dnfhv major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfeff66f-98e2-422f-a3be-cfc9f1d7a829/volumes/kubernetes.io~projected/kube-api-access-vvkf2:{mountpoint:/var/lib/kubelet/pods/cfeff66f-98e2-422f-a3be-cfc9f1d7a829/volumes/kubernetes.io~projected/kube-api-access-vvkf2 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfeff66f-98e2-422f-a3be-cfc9f1d7a829/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/cfeff66f-98e2-422f-a3be-cfc9f1d7a829/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfeff66f-98e2-422f-a3be-cfc9f1d7a829/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/cfeff66f-98e2-422f-a3be-cfc9f1d7a829/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~projected/kube-api-access-bfpxv:{mountpoint:/var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~projected/kube-api-access-bfpxv major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e57114c7-9426-4d83-870a-9d033cf335e4/volumes/kubernetes.io~projected/kube-api-access-f7h5m:{mountpoint:/var/lib/kubelet/pods/e57114c7-9426-4d83-870a-9d033cf335e4/volumes/kubernetes.io~projected/kube-api-access-f7h5m major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eef42078-d140-4216-8ee9-0cc6f0809a33/volumes/kubernetes.io~projected/kube-api-access-mm9v2:{mountpoint:/var/lib/kubelet/pods/eef42078-d140-4216-8ee9-0cc6f0809a33/volumes/kubernetes.io~projected/kube-api-access-mm9v2 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~projected/kube-api-access-bn6dd:{mountpoint:/var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~projected/kube-api-access-bn6dd major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:222 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6d09e08b3040a53a49217a28553b2bc9d06a81ac0ad346e94ee02c22096f90b9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3fe6984ec44523eb615a66bc9b1e80a43c545b15b7392624154aaf222b2b0ccf/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/28e6363bb15acbdb46f7d6edacadff819daa571bddc66c54cbaba57b26511ba5/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/720b2877bf305f2ace93b47caa616b673393cd8fccfa60277ae315dd4d044f79/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/dc12a0bb3d225d13b1710333f8b9626280af73270bcb56f4a01c6b2eb66d40e8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/035b6c93215e1c4c05040eacf486ecf09cb06d6b7a4f0fa1fdb0df4223874c00/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/78c99d4510a92f0048ea54b114fcda872d34a98db2d29ecea504a4f3b0417432/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a470008458403a999c9f2400de413898fc3a38de2e7d39f32a4153c0f686cd84/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4200b9e6f3f85e89d77c36f5f4939a390f70547aa9c37dba2a93ee00b4fe7483/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c8ce1de07d61856606250fd15d6c7fe26ff96d0a7b970248da9e839fbcfa4635/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/cadaf19d064878294e60c1f7d5bb36a141be6dc93f09e7d6e56d4d23e203fd10/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/56e870a934e63035a2d3c4525b3bf21098e4fe6bfe4278de14cdd9976d5140a8/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0cbb45b9fb63ad53d6189ad699e76b9985eab110da357b6c458108095c50dd2c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b13b91000d219c62f0f32b1e9f0e9bf2ecfa93d4f43018d93545e1c232730e29/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5d728edc69130c99bc2533efe714a40cf865d827b29de9244b84eee6940823e7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1733a42e633d61777c1b0f9ef60e988ba1d536a91d45fc1be0c47b44c06301f6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7890c6b2ec1ffd02f995a8f0bc43f77cc8ec5f10fdc5399e2783f06ad66f07f8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1daabab3b6d37cc05d4e09941328dbb8d0e8bd71db1f3ba8664b03fba63de607/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e2fb1551d7c606a1c25a645726e92f0136c043388fa95cfa82729036c7fcfe1c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b8fd0d59245e962e4ff1c1cc3b95aa2a96386e66dd2712f83f2f7f5ca6ad4eb4/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5e75638e0ead63deb2d1d9f7bdd4f5c2ead223ed6bfec179bfe3c376ea3553f6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/2a1d4b644f8abf8aa76df97c121dc96184031dc98389d7759c1b002d316d15e7/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/342392f9c7024b76ba84b4e20f8faaa1468d55b8998b293aaccb0c8c53c9a77f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/b26e0a92d1d149db7ee632a6b972453acf8497003dda0c1e3304d7ba87496a5e/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/57f5685b19b74c4ee54141d8ad0e7812def6537554ccdcb43729d74045ed221e/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/0b91de30a3490d70535a77b73b8bdd55e96045df7505e197e68b0de06828f1a6/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/988f0dd9e0de7f8288e19878351ccf5debd0bd6779b97de6fa8cb541fe8cd1fd/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/bf255dfab4db18c3351785280e729ff1a43239aea3cd28791c82ad9e44be0f95/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/f63a542e6d9a1789bb286bd5c5b50e301b44a6935ed7ff9101325430fa563cbc/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/4d303c825f430f6f0f43cf1d74b51c76c72dfd16be2936789b390ce3371a5589/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/84085ec39970ea12323f255309096d4097fe78ab9719ab081a2be3051760c057/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/8a0106e8ce4ba19ee5daa96f5176de7bbcd59a6cafe160621de0f6c7f2fbd865/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/28f5335fc215e46bb79cf99e18bd9ee219ad56cac91ddd1a614e7f1719df65ae/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/345c944deda180c9903cff2371729c17963ddc3dddcf2ffbbcd7444ef7d72414/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/0cd3eee6f16f6fbaf4787c4fb8a8e61fec18e85f124cd899eef34a8b4323bc2a/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/47bdf869141b537256ca401d3dff49ea3970b89035cc6efe7e5cd9bbe3218eec/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/78b6d8b2a6979af4e7353329f8dc3f7e39720d7ce08b6ac2c9821b74ed411c20/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/c25a68d5cc12843747b166b596cabcf112aea3539b8f0023ff6ebccfc2dc9ae5/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/8739b8ed74f3fd191311e6d2e87f06fc636448c21767a8789153899c7e53af3b/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/6cf13c43634674d627fe4a10d6554f49d4ec5e23979dd19adb365a29cf616c8e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/74fc43ff98258482478e1ff7a78eb5612a380026f2668f136d1351392486d186/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/e8264370e59fe7a13fba0f2002fd431ed9b7f4dca37985595027411de3801c01/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/835b38d7ef8f49d2c8f3ab500aef8deb83f0a26dd9ab7f4615c99c79e35fda8e/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/a2bd81bfb1a6188c9a3ae598aff4d9b5f05108703a1e202cf52e032cd7f1a20f/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/3a2217cc0e0752f72e5f045450bf50c927bd0ec1e1ffdc1eec02c9fba2a49b62/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/e35f02bbe460318815f74d3c83dc8b2c854dd785ac64906e8e464fc88cefd869/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/625ae6ec583353fec2a16414cf6d6f6c8531ebfa551da7b8381a0450c2618d5d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/03360f6f6fdb7234114fa33e4ab901a11c3f5cb4d77ddd7859040a34af49ffe6/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/0b45a7b4e3db77995237a1c1f470d5d140a113ebf77250c7f5686cf934ae3ff9/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/78259af6fd1dba2d05979ddc2782025a1b7dd195e83b3403ae4be72e2812f86c/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ab77594daf4112efe40e46c02ef58c6ee0a1fa79f1801544b1f12dd6bfaa8e99/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8333b7110eed304a3a4a2bbf2bd1c7d98267c782bd773dd78710a95d3760927f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/7fbba2c87c8e4fc71964ad674da5dc5fea51a080ec9c348d19cddac010e66688/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/178fa8b65ddc8c44a539e2ed9c8bddb63f49d698cce8da33dd11f5013f4a3a45/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.782666    4851 manager.go:217] Machine: {Timestamp:2026-11-02 01:14:54.782200359 +0000 UTC m=+0.087133176 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099359 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d33b6981b4ee9863929337168dad1 SystemUUID:ec2d33b6-981b-4ee9-8639-29337168dad1 BootID:9870b5d9-515c-4607-aeda-a15da097825e Filesystems:[{Device:/var/lib/kubelet/pods/50991da9-4137-467f-a695-0d0d4203321f/volumes/kubernetes.io~projected/kube-api-access-j7wxp DeviceMajor:0 DeviceMinor:215 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/813d9b9dc23c67b5a55eefcefb4cf6d0ebb458b12d8d8240a7085d1e8e0a9057/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc662357-7bcd-4efb-9857-37a197cb67bf/volumes/kubernetes.io~projected/kube-api-access-87d7t DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c2fae246ea9e2354173d0be93469c0dce2ae78e79e3f6f37f8d8ff5272bbdbb/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb03996-eac9-4517-9b19-283115a3ba00/volumes/kubernetes.io~projected/kube-api-access-qld9n DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4afb19d-ffe5-4011-81a0-5761db34bd75/volumes/kubernetes.io~projected/kube-api-access-cn89l DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51025d9a-739c-4934-9f5b-5bcbeb0d81ca/volumes/kubernetes.io~projected/kube-api-access-27wxr DeviceMajor:0 DeviceMinor:224 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1805ead4e559c54faff6035c9255b9a5c708195aff930e9203598fb6d5168c30/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/91f2617a-9008-4f4a-ada4-8d5997064b52/volumes/kubernetes.io~projected/kube-api-access-sbmk5 DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45ed23b73974914b493d80e358e44ab5a9678e6e26b23a8843f67a1c96a39f1c/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff3be16afe96139110677ab83296bc222bd6d027cbe69f123efe1ddd0250721b/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/975d3545-32e4-4d97-a759-d4251bb28224/volumes/kubernetes.io~projected/kube-api-access-bd26r DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:222 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51025d9a-739c-4934-9f5b-5bcbeb0d81ca/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:221 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fb18e18bdef725db03e0cdccb78cdbb10e2a4be6a9bf50facafb13b85813dee/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/414ef228-1302-4af5-8118-be23a4280494/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8d515403-65e0-4198-b78a-794ef628bc8f/volumes/kubernetes.io~projected/kube-api-access-jkrqr DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10fb972b8d11e03e3c10123b38da659288991e0786c0a985e169cc2a79c65c24/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfeff66f-98e2-422f-a3be-cfc9f1d7a829/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88c96a3e1e9db150e4c44314d2e37ddae56d98299dad5909a3baf46352e40b07/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7b6da542-d39f-4a29-9849-08164ec387e0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:269 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f/volumes/kubernetes.io~projected/kube-api-access-rwlm2 DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0ccf76ab-a4c9-4087-8ada-59b4c19218f8/volumes/kubernetes.io~projected/kube-api-access-n5whv DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83/volumes/kubernetes.io~projected/kube-api-access-t4ssf DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79b9e8663e48dfa9bec608ecd319d8aef0f9a899aa0b80e0866881a57373953b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfeff66f-98e2-422f-a3be-cfc9f1d7a829/volumes/kubernetes.io~projected/kube-api-access-vvkf2 DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6617b4113610a74824f141c8fbe72ce837d7d220b409231a44cc855c7b5a61e8/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1be28aae-f92c-4eb1-b063-f84b46967f3b/volumes/kubernetes.io~projected/kube-api-access-5fp48 DeviceMajor:0 DeviceMinor:227 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/414ef228-1302-4af5-8118-be23a4280494/volumes/kubernetes.io~projected/kube-api-access-w5krs DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91c4902be7f9730a90a271ec2de33579d88bd130fc1a04dd3db08e721e53fc09/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab7363cd-3783-4f09-b5de-ce703edde1d5/volumes/kubernetes.io~projected/kube-api-access-9wjwv DeviceMajor:0 DeviceMinor:318 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96b08fc5-cc99-4228-9a22-1e57c7fed18b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:220 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b773648e4feba6e5b8c5e28920c90ea97037e2d75222c3f8fe3d9a0244ebfe1/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3ad10cee-ccd8-424d-8e5b-8eed99b3881e/volumes/kubernetes.io~projected/kube-api-access-ztlqj DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cac6b082d49b82a25c0009cc24f65f321bf19a23b063a79ac924cc87450c3e94/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b395656d-d1a0-4cea-9b61-41788ab06732/volumes/kubernetes.io~projected/kube-api-access-pkffz DeviceMajor:0 DeviceMinor:217 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eef42078-d140-4216-8ee9-0cc6f0809a33/volumes/kubernetes.io~projected/kube-api-access-mm9v2 DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9da9a4bca090f45342cbeb031e9fd07da4ece5a9e0698a480b520c77cec098f3/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b395656d-d1a0-4cea-9b61-41788ab06732/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/16b92078-b559-44f6-85b0-4f9d1da9ddcf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4b7adf12-c300-4019-8425-6115d6ebca69/volumes/kubernetes.io~projected/kube-api-access-2pqb4 DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/68a83bf9-59f7-42a3-aeb0-efc8691a6644/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94b89615a53092de60a9fb64c3500c8748b614bcee93733bc4eee294846f1b4b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/cc662357-7bcd-4efb-9857-37a197cb67bf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes/kubernetes.io~projected/kube-api-access-bfpxv DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0dcefa6154dda1cd27789613274e478e25dbe925f881ddf2788705d8ffe2b14/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16b92078-b559-44f6-85b0-4f9d1da9ddcf/volumes/kubernetes.io~projected/kube-api-access-sg6v7 DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5d45205b-f684-4607-b594-0352c721a4f7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/68a83bf9-59f7-42a3-aeb0-efc8691a6644/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d45205b-f684-4607-b594-0352c721a4f7/volumes/kubernetes.io~projected/kube-api-access-4l4h9 DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/959d6c6c6831aca9731a1a9a29a3df47943b5c55e37372d60107742c14547efc/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes/kubernetes.io~projected/kube-api-access-bn6dd DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68a83bf9-59f7-42a3-aeb0-efc8691a6644/volumes/kubernetes.io~projected/kube-api-access-2cphg DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a924afcb201df0bdf5993b947246545049ff2a21b76e73195ea083925ca00270/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12cc7107-489a-4616-a157-e6321337b9be/volumes/kubernetes.io~projected/kube-api-access-6bq9l DeviceMajor:0 DeviceMinor:287 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f0db25b3907c838c8a9c6d0f828a8e7c18609a76007e41ac07d929963d88755/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce7ea8b0-06fd-457e-804b-da8c4f73edfc/volumes/kubernetes.io~projected/kube-api-access-dnfhv DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a74ab7ffbf0290d9cb9063a70be636b69834e9a4bfcf86496dc3078d64891b8f/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e57114c7-9426-4d83-870a-9d033cf335e4/volumes/kubernetes.io~projected/kube-api-access-f7h5m DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d45205b-f684-4607-b594-0352c721a4f7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/436b0147f53478715945f04adbd00cd09d4a2255db9854bddae8023421bdec1d/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a29820c4984e659393d0f60e28378c39745227abfecbb12e5df400a06af2bd3/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfeff66f-98e2-422f-a3be-cfc9f1d7a829/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1805ead4e559c54 MacAddress:96:10:f9:d1:5d:5a Speed:10000 Mtu:8901} {Name:436b0147f534787 MacAddress:d2:89:96:1b:13:24 Speed:10000 Mtu:8901} {Name:45ed23b73974914 MacAddress:e6:7c:d3:75:83:f5 Speed:10000 Mtu:8901} {Name:6617b4113610a74 MacAddress:22:42:94:8d:85:ff Speed:10000 Mtu:8901} {Name:6b773648e4feba6 MacAddress:4a:3c:4d:3d:b1:c7 Speed:10000 Mtu:8901} {Name:6f0db25b3907c83 MacAddress:4e:97:1c:73:39:e4 Speed:10000 Mtu:8901} {Name:813d9b9dc23c67b MacAddress:ae:4d:6d:d3:54:7d Speed:10000 Mtu:8901} {Name:88c96a3e1e9db15 MacAddress:e6:d8:9b:74:ca:39 Speed:10000 Mtu:8901} {Name:91c4902be7f9730 MacAddress:7a:da:e2:5a:b4:5b Speed:10000 Mtu:8901} {Name:a74ab7ffbf0290d MacAddress:42:bb:ef:b0:09:bb Speed:10000 Mtu:8901} {Name:a924afcb201df0b MacAddress:8a:6e:bd:15:66:43 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cb:aa:91:97 Speed:0 Mtu:9001} {Name:br-int MacAddress:fe:4d:19:48:cf:58 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:cb:aa:91:97 Speed:0 Mtu:9001} {Name:ff3be16afe96139 MacAddress:56:ec:f8:bd:4c:b5 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ba:88:94:0d:74:b2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:b9:c2:dc:71:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.782999    4851 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.783038    4851 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.783401    4851 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.783425    4851 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-23.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.783606    4851 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.783618    4851 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.783638    4851 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.783657    4851 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.783687    4851 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.783895    4851 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.784040    4851 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.784084    4851 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.784099    4851 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.784111    4851 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.784124    4851 kubelet.go:405] "Adding apiserver pod source"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.784136    4851 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.785128    4851 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.785140    4851 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:14:54.787620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.785711    4851 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 01:14:54.796549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.796522    4851 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 01:14:54.796549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.796541    4851 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:14:54.796764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.796655    4851 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 01:14:54.796906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.796891    4851 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 01:14:54.796969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.796932    4851 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 01:14:54.797059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797048    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 01:14:54.797112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797069    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 01:14:54.797112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797080    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 01:14:54.797112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797089    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 01:14:54.797112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797099    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 01:14:54.797112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797110    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 01:14:54.797366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797120    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 01:14:54.797366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797129    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 01:14:54.797366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797140    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 01:14:54.797366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797149    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 01:14:54.797366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797162    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 01:14:54.797366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797176    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 01:14:54.797366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797206    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 01:14:54.797366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.797214    4851 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 01:14:54.800850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.800832    4851 server.go:1263] "Started kubelet"
Nov 02 01:14:54.801313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.800952    4851 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 01:14:54.801313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.801022    4851 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 01:14:54.801313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.801014    4851 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 01:14:54.801505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.801344    4851 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 01:14:54.801391 ip-10-0-140-23 systemd[1]: Started Kubernetes Kubelet.
Nov 02 01:14:54.801706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.801589    4851 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:14:54.802338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.802238    4851 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 01:14:54.803616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.803595    4851 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:14:54.804284 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.804268    4851 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 01:14:54.804661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.804326    4851 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 01:14:54.804661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.804398    4851 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:09:15 +0000 UTC" deadline="2028-06-01 22:47:13.009857025 +0000 UTC"
Nov 02 01:14:54.804661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.804426    4851 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 01:14:54.804661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.804437    4851 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 01:14:54.804661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.804451    4851 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13869h32m18.205427106s"
Nov 02 01:14:54.804661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.804660    4851 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 01:14:54.809577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.809394    4851 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 01:14:54.809577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.809412    4851 factory.go:55] Registering systemd factory
Nov 02 01:14:54.809577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.809422    4851 factory.go:223] Registration of the systemd container factory successfully
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.809868    4851 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810771    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" volumeName="kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810816    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="414ef228-1302-4af5-8118-be23a4280494" volumeName="kubernetes.io/projected/414ef228-1302-4af5-8118-be23a4280494-kube-api-access-w5krs" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810838    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f2617a-9008-4f4a-ada4-8d5997064b52" volumeName="kubernetes.io/configmap/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-config" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810852    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eef42078-d140-4216-8ee9-0cc6f0809a33" volumeName="kubernetes.io/projected/eef42078-d140-4216-8ee9-0cc6f0809a33-kube-api-access-mm9v2" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810865    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb996d25-e713-4ed5-a1db-871234d5a8cd" volumeName="kubernetes.io/empty-dir/fb996d25-e713-4ed5-a1db-871234d5a8cd-ca-trust-extracted" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810885    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb996d25-e713-4ed5-a1db-871234d5a8cd" volumeName="kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-installation-pull-secrets" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810898    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab7363cd-3783-4f09-b5de-ce703edde1d5" volumeName="kubernetes.io/empty-dir/ab7363cd-3783-4f09-b5de-ce703edde1d5-data-volume" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810915    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b395656d-d1a0-4cea-9b61-41788ab06732" volumeName="kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-config" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810933    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb996d25-e713-4ed5-a1db-871234d5a8cd" volumeName="kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-bound-sa-token" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810953    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d45205b-f684-4607-b594-0352c721a4f7" volumeName="kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-etc-tuned" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810967    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d515403-65e0-4198-b78a-794ef628bc8f" volumeName="kubernetes.io/projected/8d515403-65e0-4198-b78a-794ef628bc8f-kube-api-access-jkrqr" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810980    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="975d3545-32e4-4d97-a759-d4251bb28224" volumeName="kubernetes.io/projected/975d3545-32e4-4d97-a759-d4251bb28224-kube-api-access-bd26r" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.810997    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16b92078-b559-44f6-85b0-4f9d1da9ddcf" volumeName="kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-tmp" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811027    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51025d9a-739c-4934-9f5b-5bcbeb0d81ca" volumeName="kubernetes.io/secret/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-cert" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811051    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2edcda9-40c1-4565-8843-1083f0f60b5a" volumeName="kubernetes.io/configmap/b2edcda9-40c1-4565-8843-1083f0f60b5a-nginx-conf" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811064    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f" volumeName="kubernetes.io/projected/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-kube-api-access-rwlm2" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811082    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" volumeName="kubernetes.io/projected/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-kube-api-access-ztlqj" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811097    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb03996-eac9-4517-9b19-283115a3ba00" volumeName="kubernetes.io/projected/9fb03996-eac9-4517-9b19-283115a3ba00-kube-api-access-qld9n" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811111    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d45205b-f684-4607-b594-0352c721a4f7" volumeName="kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-tmp" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811129    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1be28aae-f92c-4eb1-b063-f84b46967f3b" volumeName="kubernetes.io/projected/1be28aae-f92c-4eb1-b063-f84b46967f3b-kube-api-access-5fp48" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811141    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" volumeName="kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-binary-copy" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811160    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16b92078-b559-44f6-85b0-4f9d1da9ddcf" volumeName="kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811174    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a83bf9-59f7-42a3-aeb0-efc8691a6644" volumeName="kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-env-overrides" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811188    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab7363cd-3783-4f09-b5de-ce703edde1d5" volumeName="kubernetes.io/projected/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-api-access-9wjwv" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811206    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb996d25-e713-4ed5-a1db-871234d5a8cd" volumeName="kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811219    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d45205b-f684-4607-b594-0352c721a4f7" volumeName="kubernetes.io/projected/5d45205b-f684-4607-b594-0352c721a4f7-kube-api-access-4l4h9" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811237    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96b08fc5-cc99-4228-9a22-1e57c7fed18b" volumeName="kubernetes.io/secret/96b08fc5-cc99-4228-9a22-1e57c7fed18b-agent-certs" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811251    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce7ea8b0-06fd-457e-804b-da8c4f73edfc" volumeName="kubernetes.io/projected/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-kube-api-access-dnfhv" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811276    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e57114c7-9426-4d83-870a-9d033cf335e4" volumeName="kubernetes.io/projected/e57114c7-9426-4d83-870a-9d033cf335e4-kube-api-access-f7h5m" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811311    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb996d25-e713-4ed5-a1db-871234d5a8cd" volumeName="kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-trusted-ca" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811328    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" volumeName="kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811341    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16b92078-b559-44f6-85b0-4f9d1da9ddcf" volumeName="kubernetes.io/secret/16b92078-b559-44f6-85b0-4f9d1da9ddcf-serving-cert" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811360    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="414ef228-1302-4af5-8118-be23a4280494" volumeName="kubernetes.io/configmap/414ef228-1302-4af5-8118-be23a4280494-config" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811375    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96b08fc5-cc99-4228-9a22-1e57c7fed18b" volumeName="kubernetes.io/configmap/96b08fc5-cc99-4228-9a22-1e57c7fed18b-konnectivity-ca" seLinuxMountContext=""
Nov 02 01:14:54.811375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811390    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc662357-7bcd-4efb-9857-37a197cb67bf" volumeName="kubernetes.io/configmap/cc662357-7bcd-4efb-9857-37a197cb67bf-config" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811408    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16b92078-b559-44f6-85b0-4f9d1da9ddcf" volumeName="kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-snapshots" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811542    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a83bf9-59f7-42a3-aeb0-efc8691a6644" volumeName="kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811571    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b395656d-d1a0-4cea-9b61-41788ab06732" volumeName="kubernetes.io/projected/b395656d-d1a0-4cea-9b61-41788ab06732-kube-api-access-pkffz" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811593    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc662357-7bcd-4efb-9857-37a197cb67bf" volumeName="kubernetes.io/secret/cc662357-7bcd-4efb-9857-37a197cb67bf-serving-cert" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811606    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ccf76ab-a4c9-4087-8ada-59b4c19218f8" volumeName="kubernetes.io/projected/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-kube-api-access-n5whv" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811613    4851 factory.go:153] Registering CRI-O factory
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811624    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1be28aae-f92c-4eb1-b063-f84b46967f3b" volumeName="kubernetes.io/configmap/1be28aae-f92c-4eb1-b063-f84b46967f3b-config-volume" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811630    4851 factory.go:223] Registration of the crio container factory successfully
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811637    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1be28aae-f92c-4eb1-b063-f84b46967f3b" volumeName="kubernetes.io/empty-dir/1be28aae-f92c-4eb1-b063-f84b46967f3b-tmp-dir" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811652    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="414ef228-1302-4af5-8118-be23a4280494" volumeName="kubernetes.io/secret/414ef228-1302-4af5-8118-be23a4280494-serving-cert" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811660    4851 factory.go:103] Registering Raw factory
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811668    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a83bf9-59f7-42a3-aeb0-efc8691a6644" volumeName="kubernetes.io/projected/68a83bf9-59f7-42a3-aeb0-efc8691a6644-kube-api-access-2cphg" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811676    4851 manager.go:1199] Started watching for new ooms in manager
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811681    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4afb19d-ffe5-4011-81a0-5761db34bd75" volumeName="kubernetes.io/configmap/c4afb19d-ffe5-4011-81a0-5761db34bd75-serviceca" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811697    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e57114c7-9426-4d83-870a-9d033cf335e4" volumeName="kubernetes.io/configmap/e57114c7-9426-4d83-870a-9d033cf335e4-telemetry-config" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811711    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b7adf12-c300-4019-8425-6115d6ebca69" volumeName="kubernetes.io/projected/4b7adf12-c300-4019-8425-6115d6ebca69-kube-api-access-2pqb4" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811723    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b395656d-d1a0-4cea-9b61-41788ab06732" volumeName="kubernetes.io/secret/b395656d-d1a0-4cea-9b61-41788ab06732-serving-cert" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811740    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f" volumeName="kubernetes.io/empty-dir/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-tmp-dir" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811751    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1b792a4-f821-4dec-9db4-36bfeceee2e6" volumeName="kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-bound-sa-token" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811775    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1b792a4-f821-4dec-9db4-36bfeceee2e6" volumeName="kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811787    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1b792a4-f821-4dec-9db4-36bfeceee2e6" volumeName="kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-installation-pull-secrets" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811798    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eef42078-d140-4216-8ee9-0cc6f0809a33" volumeName="kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-daemon-config" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811815    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16b92078-b559-44f6-85b0-4f9d1da9ddcf" volumeName="kubernetes.io/projected/16b92078-b559-44f6-85b0-4f9d1da9ddcf-kube-api-access-sg6v7" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811828    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91f2617a-9008-4f4a-ada4-8d5997064b52" volumeName="kubernetes.io/projected/91f2617a-9008-4f4a-ada4-8d5997064b52-kube-api-access-sbmk5" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811844    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfeff66f-98e2-422f-a3be-cfc9f1d7a829" volumeName="kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-default-certificate" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811879    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1b792a4-f821-4dec-9db4-36bfeceee2e6" volumeName="kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-certificates" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811955    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16b92078-b559-44f6-85b0-4f9d1da9ddcf" volumeName="kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-service-ca-bundle" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811973    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f29f286-e8a8-41e8-b0b7-c18dacdbdd83" volumeName="kubernetes.io/projected/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kube-api-access-t4ssf" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.811992    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1b792a4-f821-4dec-9db4-36bfeceee2e6" volumeName="kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-trusted-ca" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812006    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1b792a4-f821-4dec-9db4-36bfeceee2e6" volumeName="kubernetes.io/empty-dir/d1b792a4-f821-4dec-9db4-36bfeceee2e6-ca-trust-extracted" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812023    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1b792a4-f821-4dec-9db4-36bfeceee2e6" volumeName="kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-kube-api-access-bfpxv" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812037    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51025d9a-739c-4934-9f5b-5bcbeb0d81ca" volumeName="kubernetes.io/projected/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-kube-api-access-27wxr" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812050    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a83bf9-59f7-42a3-aeb0-efc8691a6644" volumeName="kubernetes.io/secret/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812085    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b395656d-d1a0-4cea-9b61-41788ab06732" volumeName="kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-trusted-ca" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812096    4851 manager.go:319] Starting recovery of all containers
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812097    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfeff66f-98e2-422f-a3be-cfc9f1d7a829" volumeName="kubernetes.io/projected/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-kube-api-access-vvkf2" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812116    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb996d25-e713-4ed5-a1db-871234d5a8cd" volumeName="kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-certificates" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812130    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a83bf9-59f7-42a3-aeb0-efc8691a6644" volumeName="kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-config" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812147    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc662357-7bcd-4efb-9857-37a197cb67bf" volumeName="kubernetes.io/projected/cc662357-7bcd-4efb-9857-37a197cb67bf-kube-api-access-87d7t" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812161    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce7ea8b0-06fd-457e-804b-da8c4f73edfc" volumeName="kubernetes.io/configmap/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-iptables-alerter-script" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812176    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eef42078-d140-4216-8ee9-0cc6f0809a33" volumeName="kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-cni-binary-copy" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812193    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12cc7107-489a-4616-a157-e6321337b9be" volumeName="kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812206    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50991da9-4137-467f-a695-0d0d4203321f" volumeName="kubernetes.io/projected/50991da9-4137-467f-a695-0d0d4203321f-kube-api-access-j7wxp" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812223    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b6da542-d39f-4a29-9849-08164ec387e0" volumeName="kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812238    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4afb19d-ffe5-4011-81a0-5761db34bd75" volumeName="kubernetes.io/projected/c4afb19d-ffe5-4011-81a0-5761db34bd75-kube-api-access-cn89l" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812252    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfeff66f-98e2-422f-a3be-cfc9f1d7a829" volumeName="kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-stats-auth" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812269    4851 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb996d25-e713-4ed5-a1db-871234d5a8cd" volumeName="kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-kube-api-access-bn6dd" seLinuxMountContext=""
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812281    4851 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 01:14:54.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.812306    4851 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 01:14:54.823135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.821032    4851 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 01:14:54.828645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.828613    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/1.log"
Nov 02 01:14:54.829675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.829655    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/1.log"
Nov 02 01:14:54.836483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.836429    4851 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 01:14:54.838348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.838326    4851 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 01:14:54.838448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.838351    4851 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 01:14:54.838448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.838372    4851 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 01:14:54.838448 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:54.838411    4851 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 01:14:54.841982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.841960    4851 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:14:54.866260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.866199    4851 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="05650d2d86fa9422b4ae0f444ab8f0ddfd8446daeede737d1871a0d2dcb4b352" exitCode=0
Nov 02 01:14:54.866260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.866224    4851 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="1a74d5036266f00e100f60931147831c75cf4718f91ca24ba4aa0c3e12eacc5c" exitCode=0
Nov 02 01:14:54.866260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.866235    4851 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="f977bc985c216d036978e343c6b779dfa7a0709aedcd5f7ca14c730f9f283a7c" exitCode=0
Nov 02 01:14:54.866260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.866243    4851 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="f0c3add76a70492a448c40342f67958665cd0873310e70ffc025edc98782e6c5" exitCode=0
Nov 02 01:14:54.866260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.866250    4851 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="d31a5eaf3ae7afdb6f7732811dc4863a97a2acc97719f4bdde020087c9479015" exitCode=0
Nov 02 01:14:54.866260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.866257    4851 generic.go:356] "Generic (PLEG): container finished" podID="3ad10cee-ccd8-424d-8e5b-8eed99b3881e" containerID="7e50317c6a7f7ea04763340b1ed4701406149783494e211ddb142e3523d20dee" exitCode=0
Nov 02 01:14:54.877793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.877774    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/1.log"
Nov 02 01:14:54.877889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.877798    4851 generic.go:356] "Generic (PLEG): container finished" podID="b395656d-d1a0-4cea-9b61-41788ab06732" containerID="3c8c978e03b521b7da25f87632d3c49708401650368c5a38f93953206af92671" exitCode=255
Nov 02 01:14:54.883837 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.883811    4851 generic.go:356] "Generic (PLEG): container finished" podID="699c49bf3206a78424a5f5d5e4f1b0bc" containerID="7416ae5fe75b9d1ed86dd4570e3ac5ba4783e9b55750b54adfa6d7462b26e5ab" exitCode=0
Nov 02 01:14:54.899278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.899259    4851 manager.go:324] Recovery completed
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918467    4851 cpu_manager.go:226] "Starting" policy="none"
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918486    4851 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918507    4851 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918687    4851 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918702    4851 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918727    4851 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918735    4851 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918742    4851 policy_none.go:50] "Start"
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918762    4851 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918776    4851 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918981    4851 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.918997    4851 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 01:14:54.919184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.919005    4851 policy_none.go:44] "Start"
Nov 02 01:14:54.924032 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.924004    4851 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 01:14:54.926649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.926635    4851 manager.go:341] "Starting Device Plugin manager"
Nov 02 01:14:54.926736 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:54.926664    4851 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 01:14:54.926736 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.926682    4851 server.go:87] "Starting device plugin registration server"
Nov 02 01:14:54.927411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.927395    4851 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 01:14:54.927500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.927411    4851 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 01:14:54.927559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.927495    4851 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 01:14:54.927646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.927615    4851 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 01:14:54.927699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.927653    4851 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 01:14:54.927699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.927660    4851 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 01:14:54.927803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.927702    4851 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:14:54.927642733Z","UUID":"35555c91-ff55-4a05-8cd7-a81e9bd1fd69","Handler":null,"Name":"","Endpoint":""}
Nov 02 01:14:54.928900 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:54.928882    4851 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 01:14:54.929076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.929033    4851 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 01:14:54.929076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.929065    4851 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 01:14:54.938497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.938455    4851 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"]
Nov 02 01:14:54.939105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.939062    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal" event={"ID":"a10306333dfa6bc03e77a9c914cffe89","Type":"ContainerStarted","Data":"fd49aa6ce592ff5eeb7c7339ffd380b442507a1d83d883321a7e08e96c36620d"}
Nov 02 01:14:54.939178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.939118    4851 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:54.939178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.939123    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal" event={"ID":"a10306333dfa6bc03e77a9c914cffe89","Type":"ContainerStarted","Data":"8a29820c4984e659393d0f60e28378c39745227abfecbb12e5df400a06af2bd3"}
Nov 02 01:14:54.939178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.939163    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal" event={"ID":"699c49bf3206a78424a5f5d5e4f1b0bc","Type":"ContainerStarted","Data":"0268975cb41856a260fbe92a88124dc6cb7b457d322c2d70428330acdd4df59d"}
Nov 02 01:14:54.939178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.939170    4851 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:54.939339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.939183    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal" event={"ID":"699c49bf3206a78424a5f5d5e4f1b0bc","Type":"ContainerDied","Data":"7416ae5fe75b9d1ed86dd4570e3ac5ba4783e9b55750b54adfa6d7462b26e5ab"}
Nov 02 01:14:54.939339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.939202    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal" event={"ID":"699c49bf3206a78424a5f5d5e4f1b0bc","Type":"ContainerStarted","Data":"1c2fae246ea9e2354173d0be93469c0dce2ae78e79e3f6f37f8d8ff5272bbdbb"}
Nov 02 01:14:54.951832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.951808    4851 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:14:54.951928 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:54.951863    4851 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:54.952277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:54.952261    4851 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:14:54.952349 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:54.952319    4851 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.021892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.021860    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.021892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.021900    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.022090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.021928    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a10306333dfa6bc03e77a9c914cffe89-config\") pod \"kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"a10306333dfa6bc03e77a9c914cffe89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.027960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.027938    4851 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:14:55.030013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.029992    4851 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:14:55.030118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.030025    4851 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:14:55.030118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.030036    4851 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-23.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:14:55.030118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.030074    4851 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.042024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.042004    4851 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.042137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.042070    4851 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.122415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.122335    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.122415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.122369    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.122415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.122387    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a10306333dfa6bc03e77a9c914cffe89-config\") pod \"kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"a10306333dfa6bc03e77a9c914cffe89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.122615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.122431    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a10306333dfa6bc03e77a9c914cffe89-config\") pod \"kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"a10306333dfa6bc03e77a9c914cffe89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.122615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.122473    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.122615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.122481    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/699c49bf3206a78424a5f5d5e4f1b0bc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" (UID: \"699c49bf3206a78424a5f5d5e4f1b0bc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.785340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.785310    4851 apiserver.go:52] "Watching apiserver"
Nov 02 01:14:55.798088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.798059    4851 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 01:14:55.798799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.798766    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68","openshift-multus/multus-additional-cni-plugins-z9bzl","openshift-dns/node-resolver-2wxd5","openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw","openshift-insights/insights-operator-7c4986d449-bdwzr","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr","kube-system/konnectivity-agent-dqdrz","openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b","openshift-insights/insights-runtime-extractor-m4wnq","openshift-network-console/networking-console-plugin-86c478bb88-9fbg8","openshift-network-diagnostics/network-check-target-tkzsw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l","openshift-ingress/router-default-5c986fbdbf-f69k6","openshift-ovn-kubernetes/ovnkube-node-75mz6","kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv","openshift-console-operator/console-operator-7cd56b8dbc-cqpmz","openshift-image-registry/node-ca-hlw47","openshift-dns/dns-default-7zmf4","kube-system/kas-connection-checker-685895c7c9-jhbp5","openshift-image-registry/image-registry-7978dd58c5-947pf","kube-system/kas-connection-checker-685895c7c9-p9c2l","openshift-multus/network-metrics-daemon-2hkrs","openshift-network-operator/iptables-alerter-nk8xc","openshift-cluster-node-tuning-operator/tuned-r2dhn","openshift-image-registry/image-registry-76d8c5d8fd-g46xp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal","openshift-ingress-canary/ingress-canary-wlvr8","kube-system/kas-connection-checker-685895c7c9-5jpp9","kube-system/global-pull-secret-syncer-nm87s","openshift-multus/multus-fps5b"]
Nov 02 01:14:55.799987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.799968    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:55.800277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.800258    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.801856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.801836    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:55.802149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.802133    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.803514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.803489    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:55.807156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.806959    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.807986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.807965    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 01:14:55.808071 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.808015    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.808192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.808174    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.808327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.808200    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 01:14:55.808327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.808205    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.810192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.810175    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:55.811265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811096    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 01:14:55.811265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811143    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 01:14:55.811265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811143    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 01:14:55.811265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811174    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 01:14:55.811265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811183    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.811577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811271    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 01:14:55.811577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811303    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2zfq9\""
Nov 02 01:14:55.811577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811374    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 01:14:55.811577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811421    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8sg8n\""
Nov 02 01:14:55.811577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811461    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-btsp4\""
Nov 02 01:14:55.811577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811462    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 01:14:55.811577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811509    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 01:14:55.811577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811530    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 01:14:55.811577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811535    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wsv4x\""
Nov 02 01:14:55.811880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811584    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bjjp6\""
Nov 02 01:14:55.811880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811603    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 01:14:55.811880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811616    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 01:14:55.811880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811639    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 01:14:55.811880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811782    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zb852\""
Nov 02 01:14:55.811880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.811846    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pm2nf\""
Nov 02 01:14:55.812093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.812074    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 01:14:55.812444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.812427    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.813670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.813645    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:55.813991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.813958    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 01:14:55.814506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.814202    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 01:14:55.815357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.815336    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.815981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.815960    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:55.817853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.817820    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 01:14:55.817853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.817833    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 01:14:55.818003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.817855    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 01:14:55.818003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.817917    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.818003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.817945    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.817958    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.818003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.817984    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818013    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818042    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818064    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67z2p\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818074    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.817923    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818079    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818014    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818105    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818149    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818178    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818187    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-lq78q\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818192    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818231    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818238    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qj2bz\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818246    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818262    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818274    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818179    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818361    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818370    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818379    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818382    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818387    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jbz7p\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818283    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818414    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818413    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818417    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pl9l\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818454    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-87cgr\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818445    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818474    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818508    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818524    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818546    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818550    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7fmpb\""
Nov 02 01:14:55.818783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.818231    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.819531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819214    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.819531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819246    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.819531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819315    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.819531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819366    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.819531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819392    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7rblb\""
Nov 02 01:14:55.819748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819571    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 01:14:55.819748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819572    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-75lmf\""
Nov 02 01:14:55.819748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819682    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8rk5x\""
Nov 02 01:14:55.819748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819715    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.819748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819725    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.819922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819763    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g7hgp\""
Nov 02 01:14:55.819922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819774    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.819922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819796    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:14:55.819922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819802    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5kn9m\""
Nov 02 01:14:55.819922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819809    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.819922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819878    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 01:14:55.820092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.819931    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wfrhp\""
Nov 02 01:14:55.820092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.820037    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tmdq2\""
Nov 02 01:14:55.820092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.820076    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 01:14:55.820174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.820167    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 01:14:55.820214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.820203    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.820257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.820248    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlhpz\""
Nov 02 01:14:55.820306 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.820252    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-85fmt\""
Nov 02 01:14:55.821191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.821176    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 01:14:55.821250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.821235    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rcvqj\""
Nov 02 01:14:55.821322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.821242    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 01:14:55.821322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.821262    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 01:14:55.821322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.821305    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 01:14:55.821322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.821309    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 01:14:55.821482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.821321    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 01:14:55.821482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.821345    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6rtc\""
Nov 02 01:14:55.821482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.821397    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 01:14:55.822281 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.822269    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hdjbk\""
Nov 02 01:14:55.822379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.822368    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 01:14:55.825476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825462    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-log-socket\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.825528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825487    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-config\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.825528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825505    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ssf\" (UniqueName: \"kubernetes.io/projected/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kube-api-access-t4ssf\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.825607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825532    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7h5m\" (UniqueName: \"kubernetes.io/projected/e57114c7-9426-4d83-870a-9d033cf335e4-kube-api-access-f7h5m\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:55.825607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825577    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld9n\" (UniqueName: \"kubernetes.io/projected/9fb03996-eac9-4517-9b19-283115a3ba00-kube-api-access-qld9n\") pod \"network-check-source-6f78d59f99-cjb68\" (UID: \"9fb03996-eac9-4517-9b19-283115a3ba00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68"
Nov 02 01:14:55.825679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825653    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.825716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825681    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-config\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:55.825716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825708    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-installation-pull-secrets\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.825781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825752    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc662357-7bcd-4efb-9857-37a197cb67bf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:55.825781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825773    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd26r\" (UniqueName: \"kubernetes.io/projected/975d3545-32e4-4d97-a759-d4251bb28224-kube-api-access-bd26r\") pod \"kas-connection-checker-685895c7c9-p9c2l\" (UID: \"975d3545-32e4-4d97-a759-d4251bb28224\") " pod="kube-system/kas-connection-checker-685895c7c9-p9c2l"
Nov 02 01:14:55.825876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825792    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-stats-auth\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.825876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825817    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/414ef228-1302-4af5-8118-be23a4280494-serving-cert\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:55.825876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825840    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-service-ca-bundle\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.825876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825865    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkf2\" (UniqueName: \"kubernetes.io/projected/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-kube-api-access-vvkf2\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.826074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825877    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-config\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.826074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825887    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-netns\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.826074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825924    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-config\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:55.826074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825942    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-systemd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.826074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.825960    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc662357-7bcd-4efb-9857-37a197cb67bf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:55.826074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826053    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-stats-auth\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.826347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826072    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/414ef228-1302-4af5-8118-be23a4280494-serving-cert\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:55.826347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826098    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-netd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.826347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826125    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-installation-pull-secrets\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.826347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826135    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-script-lib\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.826347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826167    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-certificates\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.826347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826185    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-service-ca-bundle\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.826347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826271    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pqb4\" (UniqueName: \"kubernetes.io/projected/4b7adf12-c300-4019-8425-6115d6ebca69-kube-api-access-2pqb4\") pod \"volume-data-source-validator-58c9f5f46d-b4llv\" (UID: \"4b7adf12-c300-4019-8425-6115d6ebca69\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv"
Nov 02 01:14:55.826347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826327    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:55.826347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826346    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-certificates\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826357    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-socket-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826371    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovnkube-script-lib\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826384    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826411    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-default-certificate\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826483    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-ovn\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826515    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cphg\" (UniqueName: \"kubernetes.io/projected/68a83bf9-59f7-42a3-aeb0-efc8691a6644-kube-api-access-2cphg\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826530    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826542    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826582    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-registration-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826613    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-default-certificate\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826608    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-device-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826661    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-host-slash\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:55.826699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826688    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-etc-selinux\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826709    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg6v7\" (UniqueName: \"kubernetes.io/projected/16b92078-b559-44f6-85b0-4f9d1da9ddcf-kube-api-access-sg6v7\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826743    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e57114c7-9426-4d83-870a-9d033cf335e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826802    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826826    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc662357-7bcd-4efb-9857-37a197cb67bf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826843    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-slash\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826895    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-env-overrides\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826973    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb996d25-e713-4ed5-a1db-871234d5a8cd-ca-trust-extracted\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.826984    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e57114c7-9426-4d83-870a-9d033cf335e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827002    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-trusted-ca\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827012    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc662357-7bcd-4efb-9857-37a197cb67bf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827026    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/414ef228-1302-4af5-8118-be23a4280494-config\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827052    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68a83bf9-59f7-42a3-aeb0-efc8691a6644-env-overrides\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827055    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827055    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb996d25-e713-4ed5-a1db-871234d5a8cd-ca-trust-extracted\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827101    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827117    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-etc-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827142    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkffz\" (UniqueName: \"kubernetes.io/projected/b395656d-d1a0-4cea-9b61-41788ab06732-kube-api-access-pkffz\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827185    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/414ef228-1302-4af5-8118-be23a4280494-config\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827238    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.827282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827270    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn6dd\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-kube-api-access-bn6dd\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827354    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-trusted-ca\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827368    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827396    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-node-log\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827419    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b395656d-d1a0-4cea-9b61-41788ab06732-serving-cert\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827435    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-image-registry-private-configuration\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827490    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-snapshots\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827540    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-kubelet\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827565    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-var-lib-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827652    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-image-registry-private-configuration\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827672    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-snapshots\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827678    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovn-node-metrics-cert\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827725    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnfhv\" (UniqueName: \"kubernetes.io/projected/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-kube-api-access-dnfhv\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827772    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b395656d-d1a0-4cea-9b61-41788ab06732-serving-cert\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827791    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-bound-sa-token\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827825    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68a83bf9-59f7-42a3-aeb0-efc8691a6644-ovn-node-metrics-cert\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827891    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2edcda9-40c1-4565-8843-1083f0f60b5a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827908    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wxp\" (UniqueName: \"kubernetes.io/projected/50991da9-4137-467f-a695-0d0d4203321f-kube-api-access-j7wxp\") pod \"kas-connection-checker-685895c7c9-5jpp9\" (UID: \"50991da9-4137-467f-a695-0d0d4203321f\") " pod="kube-system/kas-connection-checker-685895c7c9-5jpp9"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827957    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.827989    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-sys-fs\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.828055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828044    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-iptables-alerter-script\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828083    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5krs\" (UniqueName: \"kubernetes.io/projected/414ef228-1302-4af5-8118-be23a4280494-kube-api-access-w5krs\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828102    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-tmp\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828112    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2edcda9-40c1-4565-8843-1083f0f60b5a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828124    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87d7t\" (UniqueName: \"kubernetes.io/projected/cc662357-7bcd-4efb-9857-37a197cb67bf-kube-api-access-87d7t\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828154    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrqr\" (UniqueName: \"kubernetes.io/projected/8d515403-65e0-4198-b78a-794ef628bc8f-kube-api-access-jkrqr\") pod \"kas-connection-checker-685895c7c9-jhbp5\" (UID: \"8d515403-65e0-4198-b78a-794ef628bc8f\") " pod="kube-system/kas-connection-checker-685895c7c9-jhbp5"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828170    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16b92078-b559-44f6-85b0-4f9d1da9ddcf-tmp\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828184    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmk5\" (UniqueName: \"kubernetes.io/projected/91f2617a-9008-4f4a-ada4-8d5997064b52-kube-api-access-sbmk5\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828224    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828252    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-bin\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828319    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-trusted-ca\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828363    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828407    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16b92078-b559-44f6-85b0-4f9d1da9ddcf-serving-cert\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828422    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-iptables-alerter-script\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828437    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-systemd-units\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828572    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16b92078-b559-44f6-85b0-4f9d1da9ddcf-serving-cert\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828688    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b92078-b559-44f6-85b0-4f9d1da9ddcf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.828727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.828701    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b395656d-d1a0-4cea-9b61-41788ab06732-trusted-ca\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:55.836188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.836166    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd26r\" (UniqueName: \"kubernetes.io/projected/975d3545-32e4-4d97-a759-d4251bb28224-kube-api-access-bd26r\") pod \"kas-connection-checker-685895c7c9-p9c2l\" (UID: \"975d3545-32e4-4d97-a759-d4251bb28224\") " pod="kube-system/kas-connection-checker-685895c7c9-p9c2l"
Nov 02 01:14:55.836370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.836331    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ssf\" (UniqueName: \"kubernetes.io/projected/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kube-api-access-t4ssf\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.836461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.836391    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkf2\" (UniqueName: \"kubernetes.io/projected/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-kube-api-access-vvkf2\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.836505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.836461    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7h5m\" (UniqueName: \"kubernetes.io/projected/e57114c7-9426-4d83-870a-9d033cf335e4-kube-api-access-f7h5m\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:55.836505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.836468    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld9n\" (UniqueName: \"kubernetes.io/projected/9fb03996-eac9-4517-9b19-283115a3ba00-kube-api-access-qld9n\") pod \"network-check-source-6f78d59f99-cjb68\" (UID: \"9fb03996-eac9-4517-9b19-283115a3ba00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cjb68"
Nov 02 01:14:55.836765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.836746    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pqb4\" (UniqueName: \"kubernetes.io/projected/4b7adf12-c300-4019-8425-6115d6ebca69-kube-api-access-2pqb4\") pod \"volume-data-source-validator-58c9f5f46d-b4llv\" (UID: \"4b7adf12-c300-4019-8425-6115d6ebca69\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b4llv"
Nov 02 01:14:55.836988 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.836970    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cphg\" (UniqueName: \"kubernetes.io/projected/68a83bf9-59f7-42a3-aeb0-efc8691a6644-kube-api-access-2cphg\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.837319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.837280    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn6dd\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-kube-api-access-bn6dd\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.837357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.837280    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg6v7\" (UniqueName: \"kubernetes.io/projected/16b92078-b559-44f6-85b0-4f9d1da9ddcf-kube-api-access-sg6v7\") pod \"insights-operator-7c4986d449-bdwzr\" (UID: \"16b92078-b559-44f6-85b0-4f9d1da9ddcf\") " pod="openshift-insights/insights-operator-7c4986d449-bdwzr"
Nov 02 01:14:55.840749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.840733    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkffz\" (UniqueName: \"kubernetes.io/projected/b395656d-d1a0-4cea-9b61-41788ab06732-kube-api-access-pkffz\") pod \"console-operator-7cd56b8dbc-cqpmz\" (UID: \"b395656d-d1a0-4cea-9b61-41788ab06732\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:55.843843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.843828    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrqr\" (UniqueName: \"kubernetes.io/projected/8d515403-65e0-4198-b78a-794ef628bc8f-kube-api-access-jkrqr\") pod \"kas-connection-checker-685895c7c9-jhbp5\" (UID: \"8d515403-65e0-4198-b78a-794ef628bc8f\") " pod="kube-system/kas-connection-checker-685895c7c9-jhbp5"
Nov 02 01:14:55.845069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.845049    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87d7t\" (UniqueName: \"kubernetes.io/projected/cc662357-7bcd-4efb-9857-37a197cb67bf-kube-api-access-87d7t\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-v47hr\" (UID: \"cc662357-7bcd-4efb-9857-37a197cb67bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr"
Nov 02 01:14:55.845127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.845102    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnfhv\" (UniqueName: \"kubernetes.io/projected/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-kube-api-access-dnfhv\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:55.845710 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.845694    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5krs\" (UniqueName: \"kubernetes.io/projected/414ef228-1302-4af5-8118-be23a4280494-kube-api-access-w5krs\") pod \"service-ca-operator-544d749c8c-g6jbw\" (UID: \"414ef228-1302-4af5-8118-be23a4280494\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw"
Nov 02 01:14:55.845844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.845829    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmk5\" (UniqueName: \"kubernetes.io/projected/91f2617a-9008-4f4a-ada4-8d5997064b52-kube-api-access-sbmk5\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:55.848111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.848092    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-bound-sa-token\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.849027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.849006    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wxp\" (UniqueName: \"kubernetes.io/projected/50991da9-4137-467f-a695-0d0d4203321f-kube-api-access-j7wxp\") pod \"kas-connection-checker-685895c7c9-5jpp9\" (UID: \"50991da9-4137-467f-a695-0d0d4203321f\") " pod="kube-system/kas-connection-checker-685895c7c9-5jpp9"
Nov 02 01:14:55.859117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.859059    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:55.887282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.887260    4851 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.887424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.887393    4851 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.900357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.900284    4851 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:14:55.900357 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.900340    4851 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.900802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.900790    4851 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:14:55.900842 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.900822    4851 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-23.us-east-2.compute.internal"
Nov 02 01:14:55.905544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.905529    4851 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 01:14:55.928953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.928923    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.928953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.928957    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-host-slash\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:55.929116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.928980    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5whv\" (UniqueName: \"kubernetes.io/projected/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-kube-api-access-n5whv\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:55.929116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929007    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-kubelet\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.929116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929033    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.929116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929053    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce7ea8b0-06fd-457e-804b-da8c4f73edfc-host-slash\") pod \"iptables-alerter-nk8xc\" (UID: \"ce7ea8b0-06fd-457e-804b-da8c4f73edfc\") " pod="openshift-network-operator/iptables-alerter-nk8xc"
Nov 02 01:14:55.929116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929053    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.929116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929057    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-etc-selinux\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.929322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929154    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-etc-selinux\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.929322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929207    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:55.929322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929238    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-bin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.929322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929259    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:55.929494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929329    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4h9\" (UniqueName: \"kubernetes.io/projected/5d45205b-f684-4607-b594-0352c721a4f7-kube-api-access-4l4h9\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.929494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929359    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:55.929494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929383    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.929494 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.929428    4851 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929506    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b6da542-d39f-4a29-9849-08164ec387e0-original-pull-secret\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929507    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929445    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-bound-sa-token\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.929534    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls podName:91f2617a-9008-4f4a-ada4-8d5997064b52 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:56.42951325 +0000 UTC m=+1.734446068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2v57l" (UID: "91f2617a-9008-4f4a-ada4-8d5997064b52") : secret "samples-operator-tls" not found
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929565    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-daemon-config\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929580    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929602    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-etc-tuned\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929619    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-kubelet-config\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929640    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwlm2\" (UniqueName: \"kubernetes.io/projected/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-kube-api-access-rwlm2\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929655    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-cnibin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929651    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.929666    4851 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.929682    4851 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d8c5d8fd-g46xp: secret "image-registry-tls" not found
Nov 02 01:14:55.929727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929680    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-etc-tuned\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929798    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-node-log\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.929834    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls podName:fb996d25-e713-4ed5-a1db-871234d5a8cd nodeName:}" failed. No retries permitted until 2026-11-02 01:14:56.429817159 +0000 UTC m=+1.734749990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls") pod "image-registry-76d8c5d8fd-g46xp" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd") : secret "image-registry-tls" not found
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929842    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-daemon-config\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929856    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9v2\" (UniqueName: \"kubernetes.io/projected/eef42078-d140-4216-8ee9-0cc6f0809a33-kube-api-access-mm9v2\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929857    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-node-log\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929875    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-kubelet\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929890    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-var-lib-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929934    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-var-lib-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929934    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-kubelet\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929954    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.929996    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-lib-modules\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930018    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930025    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930063    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7363cd-3783-4f09-b5de-ce703edde1d5-crio-socket\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930082    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztlqj\" (UniqueName: \"kubernetes.io/projected/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-kube-api-access-ztlqj\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930099    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-bin\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930116    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-image-registry-private-configuration\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930135    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-cni-binary-copy\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.930187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930161    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-bin\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930258    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-systemd\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930277    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-image-registry-private-configuration\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930315    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-k8s-cni-cncf-io\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930359    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eef42078-d140-4216-8ee9-0cc6f0809a33-cni-binary-copy\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930365    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4afb19d-ffe5-4011-81a0-5761db34bd75-serviceca\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930404    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930449    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930465    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-host\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930485    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn89l\" (UniqueName: \"kubernetes.io/projected/c4afb19d-ffe5-4011-81a0-5761db34bd75-kube-api-access-cn89l\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930501    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930517    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-netns\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930534    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-systemd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930575    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4afb19d-ffe5-4011-81a0-5761db34bd75-serviceca\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930579    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-conf\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930622    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-systemd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930618    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1be28aae-f92c-4eb1-b063-f84b46967f3b-config-volume\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930640    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-run-netns\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930656    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be28aae-f92c-4eb1-b063-f84b46967f3b-tmp-dir\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930715    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-system-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930741    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be28aae-f92c-4eb1-b063-f84b46967f3b-tmp-dir\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930762    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96b08fc5-cc99-4228-9a22-1e57c7fed18b-konnectivity-ca\") pod \"konnectivity-agent-dqdrz\" (UID: \"96b08fc5-cc99-4228-9a22-1e57c7fed18b\") " pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930782    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930786    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1be28aae-f92c-4eb1-b063-f84b46967f3b-config-volume\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930819    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-ovn\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930852    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-registration-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930874    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-device-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930900    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930914    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-registration-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930920    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-run-ovn\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.930918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930929    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-device-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930946    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96b08fc5-cc99-4228-9a22-1e57c7fed18b-agent-certs\") pod \"konnectivity-agent-dqdrz\" (UID: \"96b08fc5-cc99-4228-9a22-1e57c7fed18b\") " pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930961    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96b08fc5-cc99-4228-9a22-1e57c7fed18b-konnectivity-ca\") pod \"konnectivity-agent-dqdrz\" (UID: \"96b08fc5-cc99-4228-9a22-1e57c7fed18b\") " pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.930976    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-cert\") pod \"ingress-canary-wlvr8\" (UID: \"51025d9a-739c-4934-9f5b-5bcbeb0d81ca\") " pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931001    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fp48\" (UniqueName: \"kubernetes.io/projected/1be28aae-f92c-4eb1-b063-f84b46967f3b-kube-api-access-5fp48\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931025    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjwv\" (UniqueName: \"kubernetes.io/projected/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-api-access-9wjwv\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931054    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-netns\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931078    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-slash\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931145    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96b08fc5-cc99-4228-9a22-1e57c7fed18b-agent-certs\") pod \"konnectivity-agent-dqdrz\" (UID: \"96b08fc5-cc99-4228-9a22-1e57c7fed18b\") " pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931166    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-cert\") pod \"ingress-canary-wlvr8\" (UID: \"51025d9a-739c-4934-9f5b-5bcbeb0d81ca\") " pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931203    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-slash\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931257    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931279    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-trusted-ca\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931316    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpxv\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-kube-api-access-bfpxv\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931333    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.931348    4851 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.931396    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls podName:e57114c7-9426-4d83-870a-9d033cf335e4 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:56.431385718 +0000 UTC m=+1.736318522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gsw4b" (UID: "e57114c7-9426-4d83-870a-9d033cf335e4") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931350    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-socket-dir-parent\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931515    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931535    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-etc-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931551    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-var-lib-kubelet\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931574    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-sys\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931599    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-etc-openvswitch\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931600    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1b792a4-f821-4dec-9db4-36bfeceee2e6-ca-trust-extracted\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.931655    4851 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.931688    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert podName:b2edcda9-40c1-4565-8843-1083f0f60b5a nodeName:}" failed. No retries permitted until 2026-11-02 01:14:56.431679174 +0000 UTC m=+1.736611999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9fbg8" (UID: "b2edcda9-40c1-4565-8843-1083f0f60b5a") : secret "networking-console-plugin-cert" not found
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931685    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931698    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1b792a4-f821-4dec-9db4-36bfeceee2e6-ca-trust-extracted\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931730    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-kubernetes\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931748    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-dbus\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931757    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-trusted-ca\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931765    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-installation-pull-secrets\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.931759    4851 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931798    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cnibin\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.931809    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:56.431796937 +0000 UTC m=+1.736729741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : secret "router-metrics-certs-default" not found
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931838    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-sys-fs\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931860    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-modprobe-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931879    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysconfig\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931896    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wxr\" (UniqueName: \"kubernetes.io/projected/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-kube-api-access-27wxr\") pod \"ingress-canary-wlvr8\" (UID: \"51025d9a-739c-4934-9f5b-5bcbeb0d81ca\") " pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931906    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-sys-fs\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931911    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931950    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-installation-pull-secrets\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931971    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-conf-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.931997    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-systemd-units\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932019    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-log-socket\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.932013 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:55.932032    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:56.432022124 +0000 UTC m=+1.736954937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : configmap references non-existent config key: service-ca.crt
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932053    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7363cd-3783-4f09-b5de-ce703edde1d5-data-volume\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932055    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-log-socket\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932073    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-run\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932094    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-tmp\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932110    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7363cd-3783-4f09-b5de-ce703edde1d5-data-volume\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932115    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-systemd-units\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932140    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-multus\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932155    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d45205b-f684-4607-b594-0352c721a4f7-tmp\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932169    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-netd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932189    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-certificates\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932234    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-tmp-dir\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932277    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-certificates\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932299    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-cni-netd\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932316    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932330    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-tmp-dir\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932332    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932365    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4afb19d-ffe5-4011-81a0-5761db34bd75-host\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932400    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-socket-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932434    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f29f286-e8a8-41e8-b0b7-c18dacdbdd83-socket-dir\") pod \"aws-ebs-csi-driver-node-dw7wh\" (UID: \"5f29f286-e8a8-41e8-b0b7-c18dacdbdd83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932455    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-hosts-file\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932489    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-multus-certs\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932532    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-etc-kubernetes\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932572    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-system-cni-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932600    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932632    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932655    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-hostroot\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932678    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68a83bf9-59f7-42a3-aeb0-efc8691a6644-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75mz6\" (UID: \"68a83bf9-59f7-42a3-aeb0-efc8691a6644\") " pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932681    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.933541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.932792    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.940788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.940762    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5whv\" (UniqueName: \"kubernetes.io/projected/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-kube-api-access-n5whv\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:55.940963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.940945    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4h9\" (UniqueName: \"kubernetes.io/projected/5d45205b-f684-4607-b594-0352c721a4f7-kube-api-access-4l4h9\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:55.941021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.940980    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-bound-sa-token\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:55.941499 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.941480    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9v2\" (UniqueName: \"kubernetes.io/projected/eef42078-d140-4216-8ee9-0cc6f0809a33-kube-api-access-mm9v2\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:55.941544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.941482    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwlm2\" (UniqueName: \"kubernetes.io/projected/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-kube-api-access-rwlm2\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:55.943110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.943096    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn89l\" (UniqueName: \"kubernetes.io/projected/c4afb19d-ffe5-4011-81a0-5761db34bd75-kube-api-access-cn89l\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:55.943394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.943379    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fp48\" (UniqueName: \"kubernetes.io/projected/1be28aae-f92c-4eb1-b063-f84b46967f3b-kube-api-access-5fp48\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:55.943873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.943858    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztlqj\" (UniqueName: \"kubernetes.io/projected/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-kube-api-access-ztlqj\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:55.947067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.947045    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjwv\" (UniqueName: \"kubernetes.io/projected/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-api-access-9wjwv\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:55.947242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.947226    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bq9l\" (UniqueName: \"kubernetes.io/projected/12cc7107-489a-4616-a157-e6321337b9be-kube-api-access-6bq9l\") pod \"network-check-target-tkzsw\" (UID: \"12cc7107-489a-4616-a157-e6321337b9be\") " pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:14:55.947395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.947377    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wxr\" (UniqueName: \"kubernetes.io/projected/51025d9a-739c-4934-9f5b-5bcbeb0d81ca-kube-api-access-27wxr\") pod \"ingress-canary-wlvr8\" (UID: \"51025d9a-739c-4934-9f5b-5bcbeb0d81ca\") " pod="openshift-ingress-canary/ingress-canary-wlvr8"
Nov 02 01:14:55.947444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:55.947431    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpxv\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-kube-api-access-bfpxv\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:56.033527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033495    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:56.033527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033533    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-kubelet-config\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:56.033738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033549    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-cnibin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.033738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033612    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-cnibin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.033738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033636    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-lib-modules\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.033738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033641    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-kubelet-config\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:56.033738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033656    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:56.033738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033681    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7363cd-3783-4f09-b5de-ce703edde1d5-crio-socket\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:56.033738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033707    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-systemd\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.033738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033724    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-k8s-cni-cncf-io\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033744    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-host\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033774    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7363cd-3783-4f09-b5de-ce703edde1d5-crio-socket\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033784    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-lib-modules\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033794    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-host\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033788    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-systemd\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033803    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-k8s-cni-cncf-io\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033855    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.033857    4851 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033885    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-conf\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033907    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-system-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.033918    4851 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033928    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.033960    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:56.533942161 +0000 UTC m=+1.838874987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : secret "metrics-daemon-secret" not found
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033976    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-system-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.033988    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-cni-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.034031    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls podName:1be28aae-f92c-4eb1-b063-f84b46967f3b nodeName:}" failed. No retries permitted until 2026-11-02 01:14:56.534015091 +0000 UTC m=+1.838947898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls") pod "dns-default-7zmf4" (UID: "1be28aae-f92c-4eb1-b063-f84b46967f3b") : secret "dns-default-metrics-tls" not found
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034050    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-conf\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034061    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034093    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-netns\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034097    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7363cd-3783-4f09-b5de-ce703edde1d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034145    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-netns\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034177    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034203    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-socket-dir-parent\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034218    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysctl-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034243    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-socket-dir-parent\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.034261    4851 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.034317    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:56.534282902 +0000 UTC m=+1.839215710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034339    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-var-lib-kubelet\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034355    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-sys\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034390    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-kubernetes\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034414    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-dbus\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034418    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-var-lib-kubelet\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034438    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cnibin\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034465    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-modprobe-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034489    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysconfig\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034496    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-cnibin\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034500    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b6da542-d39f-4a29-9849-08164ec387e0-dbus\") pod \"global-pull-secret-syncer-nm87s\" (UID: \"7b6da542-d39f-4a29-9849-08164ec387e0\") " pod="kube-system/global-pull-secret-syncer-nm87s"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034466    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-kubernetes\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034498    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-sys\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034524    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-conf-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034538    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-sysconfig\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034549    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-multus-conf-dir\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034557    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-run\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034564    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-etc-modprobe-d\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034585    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-multus\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034615    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034638    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4afb19d-ffe5-4011-81a0-5761db34bd75-host\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034646    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d45205b-f684-4607-b594-0352c721a4f7-run\") pod \"tuned-r2dhn\" (UID: \"5d45205b-f684-4607-b594-0352c721a4f7\") " pod="openshift-cluster-node-tuning-operator/tuned-r2dhn"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034657    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-multus\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034665    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-hosts-file\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034669    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034699    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f-hosts-file\") pod \"node-resolver-2wxd5\" (UID: \"b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f\") " pod="openshift-dns/node-resolver-2wxd5"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034673    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4afb19d-ffe5-4011-81a0-5761db34bd75-host\") pod \"node-ca-hlw47\" (UID: \"c4afb19d-ffe5-4011-81a0-5761db34bd75\") " pod="openshift-image-registry/node-ca-hlw47"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034696    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-multus-certs\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034730    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-etc-kubernetes\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034734    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-run-multus-certs\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034745    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-system-cni-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034762    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-etc-kubernetes\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034776    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ad10cee-ccd8-424d-8e5b-8eed99b3881e-system-cni-dir\") pod \"multus-additional-cni-plugins-z9bzl\" (UID: \"3ad10cee-ccd8-424d-8e5b-8eed99b3881e\") " pod="openshift-multus/multus-additional-cni-plugins-z9bzl"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034795    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034810    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-hostroot\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034827    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-kubelet\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034843    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-bin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034860    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-hostroot\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034887    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-cni-bin\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.034903    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef42078-d140-4216-8ee9-0cc6f0809a33-host-var-lib-kubelet\") pod \"multus-fps5b\" (UID: \"eef42078-d140-4216-8ee9-0cc6f0809a33\") " pod="openshift-multus/multus-fps5b"
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.034913    4851 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.034922    4851 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7978dd58c5-947pf: secret "image-registry-tls" not found
Nov 02 01:14:56.034948 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.034963    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls podName:d1b792a4-f821-4dec-9db4-36bfeceee2e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:56.534954345 +0000 UTC m=+1.839887148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls") pod "image-registry-7978dd58c5-947pf" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6") : secret "image-registry-tls" not found
Nov 02 01:14:56.102913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.102888    4851 scope.go:122] "RemoveContainer" containerID="3c8c978e03b521b7da25f87632d3c49708401650368c5a38f93953206af92671"
Nov 02 01:14:56.118004 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.117971    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:56.241648 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.241619    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:56.438792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.438705    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:56.438792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.438752    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:56.438792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.438794    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:56.439030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.438813    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:56.439030 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.438867    4851 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:14:56.439030 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.438932    4851 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:14:56.439030 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.438938    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.438917299 +0000 UTC m=+2.743850104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : configmap references non-existent config key: service-ca.crt
Nov 02 01:14:56.439030 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.438954    4851 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:56.439030 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.438967    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.438956139 +0000 UTC m=+2.743888947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : secret "router-metrics-certs-default" not found
Nov 02 01:14:56.439030 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.438968    4851 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:56.439030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.438872    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:56.439030 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.438984    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls podName:91f2617a-9008-4f4a-ada4-8d5997064b52 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.43897521 +0000 UTC m=+2.743908015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2v57l" (UID: "91f2617a-9008-4f4a-ada4-8d5997064b52") : secret "samples-operator-tls" not found
Nov 02 01:14:56.439030 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.439009    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls podName:e57114c7-9426-4d83-870a-9d033cf335e4 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.438999557 +0000 UTC m=+2.743932361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gsw4b" (UID: "e57114c7-9426-4d83-870a-9d033cf335e4") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:56.439358 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.438969    4851 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d8c5d8fd-g46xp: secret "image-registry-tls" not found
Nov 02 01:14:56.439358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.439040    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:56.439358 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.439078    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls podName:fb996d25-e713-4ed5-a1db-871234d5a8cd nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.439068071 +0000 UTC m=+2.744000874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls") pod "image-registry-76d8c5d8fd-g46xp" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd") : secret "image-registry-tls" not found
Nov 02 01:14:56.439358 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.439103    4851 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:14:56.439358 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.439136    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert podName:b2edcda9-40c1-4565-8843-1083f0f60b5a nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.439128352 +0000 UTC m=+2.744061181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9fbg8" (UID: "b2edcda9-40c1-4565-8843-1083f0f60b5a") : secret "networking-console-plugin-cert" not found
Nov 02 01:14:56.539460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.539428    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:56.539611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.539487    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:56.539611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.539524    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:56.539611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.539565    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:56.539611 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.539572    4851 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:56.539611 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.539591    4851 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7978dd58c5-947pf: secret "image-registry-tls" not found
Nov 02 01:14:56.539788 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.539641    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls podName:d1b792a4-f821-4dec-9db4-36bfeceee2e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.539627589 +0000 UTC m=+2.844560407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls") pod "image-registry-7978dd58c5-947pf" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6") : secret "image-registry-tls" not found
Nov 02 01:14:56.539788 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.539657    4851 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:14:56.539788 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.539657    4851 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:56.539788 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.539709    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.539694934 +0000 UTC m=+2.844627743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:56.539788 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.539723    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.539716118 +0000 UTC m=+2.844648923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : secret "metrics-daemon-secret" not found
Nov 02 01:14:56.539788 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.539734    4851 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:14:56.539788 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.539779    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls podName:1be28aae-f92c-4eb1-b063-f84b46967f3b nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.539767627 +0000 UTC m=+2.844700434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls") pod "dns-default-7zmf4" (UID: "1be28aae-f92c-4eb1-b063-f84b46967f3b") : secret "dns-default-metrics-tls" not found
Nov 02 01:14:56.892344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.892177    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:14:56.893850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.892951    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/1.log"
Nov 02 01:14:56.893850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.892990    4851 generic.go:356] "Generic (PLEG): container finished" podID="b395656d-d1a0-4cea-9b61-41788ab06732" containerID="eecf329e7a09aa64bb5fb655a875e0793c8517dcfed172f332ff25728a479421" exitCode=255
Nov 02 01:14:56.893850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.893797    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" event={"ID":"b395656d-d1a0-4cea-9b61-41788ab06732","Type":"ContainerDied","Data":"eecf329e7a09aa64bb5fb655a875e0793c8517dcfed172f332ff25728a479421"}
Nov 02 01:14:56.894120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.893962    4851 scope.go:122] "RemoveContainer" containerID="3c8c978e03b521b7da25f87632d3c49708401650368c5a38f93953206af92671"
Nov 02 01:14:56.894671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:56.894652    4851 scope.go:122] "RemoveContainer" containerID="eecf329e7a09aa64bb5fb655a875e0793c8517dcfed172f332ff25728a479421"
Nov 02 01:14:56.894954 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:56.894927    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cqpmz_openshift-console-operator(b395656d-d1a0-4cea-9b61-41788ab06732)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732"
Nov 02 01:14:57.449337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.449256    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:57.449540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.449356    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:57.449540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.449386    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:57.449540 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449421    4851 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:57.449540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.449424    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:57.449540 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449487    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls podName:e57114c7-9426-4d83-870a-9d033cf335e4 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.449470796 +0000 UTC m=+4.754403601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gsw4b" (UID: "e57114c7-9426-4d83-870a-9d033cf335e4") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:57.449540 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449509    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.449499657 +0000 UTC m=+4.754432460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : configmap references non-existent config key: service-ca.crt
Nov 02 01:14:57.449540 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449510    4851 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:14:57.449820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.449565    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:57.449820 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449573    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert podName:b2edcda9-40c1-4565-8843-1083f0f60b5a nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.449559898 +0000 UTC m=+4.754492703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9fbg8" (UID: "b2edcda9-40c1-4565-8843-1083f0f60b5a") : secret "networking-console-plugin-cert" not found
Nov 02 01:14:57.449820 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449510    4851 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 01:14:57.449820 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449626    4851 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:14:57.449820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.449606    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:57.449820 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449658    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls podName:91f2617a-9008-4f4a-ada4-8d5997064b52 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.449647737 +0000 UTC m=+4.754580563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2v57l" (UID: "91f2617a-9008-4f4a-ada4-8d5997064b52") : secret "samples-operator-tls" not found
Nov 02 01:14:57.449820 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449659    4851 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:57.449820 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449684    4851 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76d8c5d8fd-g46xp: secret "image-registry-tls" not found
Nov 02 01:14:57.449820 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449710    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs podName:cfeff66f-98e2-422f-a3be-cfc9f1d7a829 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.449690501 +0000 UTC m=+4.754623308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs") pod "router-default-5c986fbdbf-f69k6" (UID: "cfeff66f-98e2-422f-a3be-cfc9f1d7a829") : secret "router-metrics-certs-default" not found
Nov 02 01:14:57.449820 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.449733    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls podName:fb996d25-e713-4ed5-a1db-871234d5a8cd nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.449726191 +0000 UTC m=+4.754658998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls") pod "image-registry-76d8c5d8fd-g46xp" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd") : secret "image-registry-tls" not found
Nov 02 01:14:57.550517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.550480    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:57.550702 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.550540    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:57.550702 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.550565    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:57.550702 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.550601    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:57.550702 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.550631    4851 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 01:14:57.550702 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.550650    4851 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7978dd58c5-947pf: secret "image-registry-tls" not found
Nov 02 01:14:57.550702 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.550698    4851 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:57.550893 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.550704    4851 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:14:57.550893 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.550704    4851 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:14:57.550893 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.550714    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls podName:d1b792a4-f821-4dec-9db4-36bfeceee2e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.550700356 +0000 UTC m=+4.855633160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls") pod "image-registry-7978dd58c5-947pf" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6") : secret "image-registry-tls" not found
Nov 02 01:14:57.550893 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.550778    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.55076358 +0000 UTC m=+4.855696384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:57.550893 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.550794    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.550785261 +0000 UTC m=+4.855718092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : secret "metrics-daemon-secret" not found
Nov 02 01:14:57.550893 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:57.550804    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls podName:1be28aae-f92c-4eb1-b063-f84b46967f3b nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.550798957 +0000 UTC m=+4.855731762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls") pod "dns-default-7zmf4" (UID: "1be28aae-f92c-4eb1-b063-f84b46967f3b") : secret "dns-default-metrics-tls" not found
Nov 02 01:14:57.897424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.897343    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:14:57.909303 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:57.909261    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:58.044475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:58.044443    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:58.172231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:58.172203    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:58.416127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:58.416099    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:58.558254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:58.558177    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:58.811160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:58.811077    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:59.402689 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.402651    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:59.468535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.468494    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:14:59.468677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.468557    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:14:59.468677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.468598    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:59.468677 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.468632    4851 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:59.468677 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.468641    4851 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:14:59.468882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.468633    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:59.468882 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.468691    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls podName:e57114c7-9426-4d83-870a-9d033cf335e4 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:03.468671807 +0000 UTC m=+8.773604612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gsw4b" (UID: "e57114c7-9426-4d83-870a-9d033cf335e4") : secret "cluster-monitoring-operator-tls" not found
Nov 02 01:14:59.468882 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.468768    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert podName:b2edcda9-40c1-4565-8843-1083f0f60b5a nodeName:}" failed. No retries permitted until 2026-11-02 01:15:03.468755787 +0000 UTC m=+8.773688592 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9fbg8" (UID: "b2edcda9-40c1-4565-8843-1083f0f60b5a") : secret "networking-console-plugin-cert" not found
Nov 02 01:14:59.468882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.468829    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:14:59.468882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.468854    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:59.469152 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.468954    4851 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:14:59.469152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.468940    4851 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 01:14:59.469152 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.469000    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls podName:91f2617a-9008-4f4a-ada4-8d5997064b52 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:03.468985943 +0000 UTC m=+8.773918764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2v57l" (UID: "91f2617a-9008-4f4a-ada4-8d5997064b52") : secret "samples-operator-tls" not found
Nov 02 01:14:59.469394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.469376    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-service-ca-bundle\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:59.471254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.471239    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfeff66f-98e2-422f-a3be-cfc9f1d7a829-metrics-certs\") pod \"router-default-5c986fbdbf-f69k6\" (UID: \"cfeff66f-98e2-422f-a3be-cfc9f1d7a829\") " pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:59.471348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.471330    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"image-registry-76d8c5d8fd-g46xp\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") " pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:59.531256 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.531233    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:59.569365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.569345    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:14:59.569428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.569391    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:59.569484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.569428    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:14:59.569484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.569451    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:14:59.569484 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.569469    4851 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:59.569580 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.569513    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:03.569501116 +0000 UTC m=+8.874433920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:14:59.569580 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.569529    4851 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:14:59.569580 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.569567    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:03.56955441 +0000 UTC m=+8.874487215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : secret "metrics-daemon-secret" not found
Nov 02 01:14:59.569689 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.569604    4851 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:14:59.569689 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.569636    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls podName:1be28aae-f92c-4eb1-b063-f84b46967f3b nodeName:}" failed. No retries permitted until 2026-11-02 01:15:03.569626899 +0000 UTC m=+8.874559703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls") pod "dns-default-7zmf4" (UID: "1be28aae-f92c-4eb1-b063-f84b46967f3b") : secret "dns-default-metrics-tls" not found
Nov 02 01:14:59.571704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.571686    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"image-registry-7978dd58c5-947pf\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") " pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:59.661407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.661384    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:59.701109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.701085    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:14:59.702861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.702844    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:14:59.713600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.713580    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:14:59.771475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.768916    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:14:59.771475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.769361    4851 scope.go:122] "RemoveContainer" containerID="eecf329e7a09aa64bb5fb655a875e0793c8517dcfed172f332ff25728a479421"
Nov 02 01:14:59.771475 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:14:59.769600    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cqpmz_openshift-console-operator(b395656d-d1a0-4cea-9b61-41788ab06732)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732"
Nov 02 01:14:59.820505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.820475    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:14:59.983454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.983391    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c986fbdbf-f69k6"]
Nov 02 01:14:59.987826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.987743    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76d8c5d8fd-g46xp"]
Nov 02 01:14:59.988644 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:59.988607    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfeff66f_98e2_422f_a3be_cfc9f1d7a829.slice/crio-ea6b7445e43c78eab0d39d3b48e24d7ed9abea5c2896c42261f130a2239c9032 WatchSource:0}: Error finding container ea6b7445e43c78eab0d39d3b48e24d7ed9abea5c2896c42261f130a2239c9032: Status 404 returned error can't find the container with id ea6b7445e43c78eab0d39d3b48e24d7ed9abea5c2896c42261f130a2239c9032
Nov 02 01:14:59.992265 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:59.992244    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb996d25_e713_4ed5_a1db_871234d5a8cd.slice/crio-4b2123c0e16336065bbba150c11fc2182b294dd71bedb906b4d208328d374eb5 WatchSource:0}: Error finding container 4b2123c0e16336065bbba150c11fc2182b294dd71bedb906b4d208328d374eb5: Status 404 returned error can't find the container with id 4b2123c0e16336065bbba150c11fc2182b294dd71bedb906b4d208328d374eb5
Nov 02 01:14:59.997337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:14:59.997314    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7978dd58c5-947pf"]
Nov 02 01:15:00.000015 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:14:59.999993    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b792a4_f821_4dec_9db4_36bfeceee2e6.slice/crio-9acc222948a3d1cea2f0edf0bd794d8d4b8889f082defba66de4e258cdb72602 WatchSource:0}: Error finding container 9acc222948a3d1cea2f0edf0bd794d8d4b8889f082defba66de4e258cdb72602: Status 404 returned error can't find the container with id 9acc222948a3d1cea2f0edf0bd794d8d4b8889f082defba66de4e258cdb72602
Nov 02 01:15:00.011666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.011635    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:00.113864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.113707    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:15:00.200749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.200708    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:00.213709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.213669    4851 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:15:00.214086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.214070    4851 scope.go:122] "RemoveContainer" containerID="eecf329e7a09aa64bb5fb655a875e0793c8517dcfed172f332ff25728a479421"
Nov 02 01:15:00.214284 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:00.214264    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cqpmz_openshift-console-operator(b395656d-d1a0-4cea-9b61-41788ab06732)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732"
Nov 02 01:15:00.799887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.799857    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:00.908033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.908003    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7978dd58c5-947pf" event={"ID":"d1b792a4-f821-4dec-9db4-36bfeceee2e6","Type":"ContainerStarted","Data":"42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b"}
Nov 02 01:15:00.908207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.908048    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7978dd58c5-947pf" event={"ID":"d1b792a4-f821-4dec-9db4-36bfeceee2e6","Type":"ContainerStarted","Data":"9acc222948a3d1cea2f0edf0bd794d8d4b8889f082defba66de4e258cdb72602"}
Nov 02 01:15:00.909541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.909515    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c986fbdbf-f69k6" event={"ID":"cfeff66f-98e2-422f-a3be-cfc9f1d7a829","Type":"ContainerStarted","Data":"27f4686c2fb6988c8c88bf5f39915da2d911134115dd2088c9cdb48181d197fa"}
Nov 02 01:15:00.909672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.909562    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c986fbdbf-f69k6" event={"ID":"cfeff66f-98e2-422f-a3be-cfc9f1d7a829","Type":"ContainerStarted","Data":"ea6b7445e43c78eab0d39d3b48e24d7ed9abea5c2896c42261f130a2239c9032"}
Nov 02 01:15:00.910896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.910878    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp" event={"ID":"fb996d25-e713-4ed5-a1db-871234d5a8cd","Type":"ContainerStarted","Data":"035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f"}
Nov 02 01:15:00.910987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:00.910911    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp" event={"ID":"fb996d25-e713-4ed5-a1db-871234d5a8cd","Type":"ContainerStarted","Data":"4b2123c0e16336065bbba150c11fc2182b294dd71bedb906b4d208328d374eb5"}
Nov 02 01:15:01.219315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:01.219269    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:15:01.219488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:01.219476    4851 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:15:01.409375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:01.409340    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:01.450455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:01.450406    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:15:01.601955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:01.601873    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:01.703124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:01.703088    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:15:01.705707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:01.705686    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:15:01.914478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:01.914450    4851 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:15:01.914942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:01.914916    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:15:01.915831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:01.915817    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c986fbdbf-f69k6"
Nov 02 01:15:02.008359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:02.008332    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:02.209581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:02.209491    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:02.810700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:02.810661    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:03.414388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.414361    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:03.509619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.509587    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:15:03.509770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.509627    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:15:03.509770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.509665    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:15:03.509770 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:03.509752    4851 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 01:15:03.509870 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:03.509792    4851 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 01:15:03.509870 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:03.509818    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls podName:91f2617a-9008-4f4a-ada4-8d5997064b52 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:11.509805071 +0000 UTC m=+16.814737876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2v57l" (UID: "91f2617a-9008-4f4a-ada4-8d5997064b52") : secret "samples-operator-tls" not found
Nov 02 01:15:03.509870 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:03.509850    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert podName:b2edcda9-40c1-4565-8843-1083f0f60b5a nodeName:}" failed. No retries permitted until 2026-11-02 01:15:11.509837394 +0000 UTC m=+16.814770198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9fbg8" (UID: "b2edcda9-40c1-4565-8843-1083f0f60b5a") : secret "networking-console-plugin-cert" not found
Nov 02 01:15:03.512245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.512221    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e57114c7-9426-4d83-870a-9d033cf335e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gsw4b\" (UID: \"e57114c7-9426-4d83-870a-9d033cf335e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:15:03.579679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.579655    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2wxd5_b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f/dns-node-resolver/0.log"
Nov 02 01:15:03.602921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.602898    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"
Nov 02 01:15:03.610987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.610964    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:15:03.611138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.611002    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:15:03.611138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.611065    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:15:03.611138 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:03.611093    4851 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 01:15:03.611331 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:03.611162    4851 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:15:03.611331 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:03.611162    4851 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 01:15:03.611331 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:03.611173    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls podName:1be28aae-f92c-4eb1-b063-f84b46967f3b nodeName:}" failed. No retries permitted until 2026-11-02 01:15:11.611142966 +0000 UTC m=+16.916075788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls") pod "dns-default-7zmf4" (UID: "1be28aae-f92c-4eb1-b063-f84b46967f3b") : secret "dns-default-metrics-tls" not found
Nov 02 01:15:03.611331 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:03.611205    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs podName:0ccf76ab-a4c9-4087-8ada-59b4c19218f8 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:11.611191655 +0000 UTC m=+16.916124472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs") pod "network-metrics-daemon-2hkrs" (UID: "0ccf76ab-a4c9-4087-8ada-59b4c19218f8") : secret "metrics-daemon-secret" not found
Nov 02 01:15:03.611331 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:03.611221    4851 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls podName:ab7363cd-3783-4f09-b5de-ce703edde1d5 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:11.611212759 +0000 UTC m=+16.916145563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4wnq" (UID: "ab7363cd-3783-4f09-b5de-ce703edde1d5") : secret "insights-runtime-extractor-tls" not found
Nov 02 01:15:03.612069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.612049    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:03.698944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.698881    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:15:03.901212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.901184    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b"]
Nov 02 01:15:03.904455 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:15:03.904424    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57114c7_9426_4d83_870a_9d033cf335e4.slice/crio-d9948d97e6c12e09b8368c4f65ee4ff897c2b354746640a4631837a1f6e23710 WatchSource:0}: Error finding container d9948d97e6c12e09b8368c4f65ee4ff897c2b354746640a4631837a1f6e23710: Status 404 returned error can't find the container with id d9948d97e6c12e09b8368c4f65ee4ff897c2b354746640a4631837a1f6e23710
Nov 02 01:15:03.906600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.906584    4851 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:15:03.922479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:03.922452    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b" event={"ID":"e57114c7-9426-4d83-870a-9d033cf335e4","Type":"ContainerStarted","Data":"d9948d97e6c12e09b8368c4f65ee4ff897c2b354746640a4631837a1f6e23710"}
Nov 02 01:15:04.010710 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:04.010635    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:04.181283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:04.181255    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-76d8c5d8fd-g46xp_fb996d25-e713-4ed5-a1db-871234d5a8cd/registry/0.log"
Nov 02 01:15:04.206951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:04.206926    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dqdrz"
Nov 02 01:15:04.210567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:04.210536    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:04.380111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:04.380039    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7978dd58c5-947pf_d1b792a4-f821-4dec-9db4-36bfeceee2e6/registry/0.log"
Nov 02 01:15:04.780386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:04.780359    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlw47_c4afb19d-ffe5-4011-81a0-5761db34bd75/node-ca/0.log"
Nov 02 01:15:04.813200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:04.813172    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:04.901813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:04.901790    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:15:05.066191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:05.066105    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75mz6"
Nov 02 01:15:05.182635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:05.182606    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c986fbdbf-f69k6_cfeff66f-98e2-422f-a3be-cfc9f1d7a829/router/0.log"
Nov 02 01:15:05.410865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:05.410786    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:05.613102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:05.613071    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:05.779666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:05.779602    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wlvr8_51025d9a-739c-4934-9f5b-5bcbeb0d81ca/serve-healthcheck-canary/0.log"
Nov 02 01:15:06.008579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:06.008541    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:06.214726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:06.214689    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:06.381301 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:06.381255    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-v47hr_cc662357-7bcd-4efb-9857-37a197cb67bf/kube-storage-version-migrator-operator/0.log"
Nov 02 01:15:06.748644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:06.748617    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:06.882056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:06.882019    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:06.934370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:06.934334    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gsw4b" event={"ID":"e57114c7-9426-4d83-870a-9d033cf335e4","Type":"ContainerStarted","Data":"8b06b67a8d767b32a473262305a355eee2a1ca50aaca63bd154d368c54d0e877"}
Nov 02 01:15:07.022465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:07.022384    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:07.414381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:07.414353    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:07.601716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:07.601684    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:07.839678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:07.839608    4851 scope.go:122] "RemoveContainer" containerID="eecf329e7a09aa64bb5fb655a875e0793c8517dcfed172f332ff25728a479421"
Nov 02 01:15:07.939547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:07.939522    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:15:07.939706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:07.939634    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" event={"ID":"b395656d-d1a0-4cea-9b61-41788ab06732","Type":"ContainerStarted","Data":"a8839f43fe135a846a7f186fc6882392f19b0866e92746c0c3110782e092cf50"}
Nov 02 01:15:07.939917 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:07.939897    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:15:07.940820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:07.940797    4851 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-cqpmz container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body=
Nov 02 01:15:07.940920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:07.940848    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused"
Nov 02 01:15:08.006008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:08.005975    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:08.201341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:08.201281    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:08.460711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:08.460615    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:08.594541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:08.594515    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:08.726533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:08.726468    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:08.854748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:08.854717    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:08.947172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:08.947146    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz"
Nov 02 01:15:09.086629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:09.086540    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:09.211918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:09.211878    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:11.113081 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.113047    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:11.242988 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.242956    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:11.588193 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.588154    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:15:11.588406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.588222    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:15:11.590915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.590885    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91f2617a-9008-4f4a-ada4-8d5997064b52-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2v57l\" (UID: \"91f2617a-9008-4f4a-ada4-8d5997064b52\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:15:11.591026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.590918    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2edcda9-40c1-4565-8843-1083f0f60b5a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9fbg8\" (UID: \"b2edcda9-40c1-4565-8843-1083f0f60b5a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:15:11.598111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.598078    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:11.689413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.689384    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:15:11.689413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.689424    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:15:11.689631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.689456    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:15:11.692046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.692022    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ccf76ab-a4c9-4087-8ada-59b4c19218f8-metrics-certs\") pod \"network-metrics-daemon-2hkrs\" (UID: \"0ccf76ab-a4c9-4087-8ada-59b4c19218f8\") " pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:15:11.692157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.692116    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be28aae-f92c-4eb1-b063-f84b46967f3b-metrics-tls\") pod \"dns-default-7zmf4\" (UID: \"1be28aae-f92c-4eb1-b063-f84b46967f3b\") " pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:15:11.692157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.692137    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7363cd-3783-4f09-b5de-ce703edde1d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4wnq\" (UID: \"ab7363cd-3783-4f09-b5de-ce703edde1d5\") " pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:15:11.701069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.701052    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"
Nov 02 01:15:11.704268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.704255    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"
Nov 02 01:15:11.711341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.711324    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hkrs"
Nov 02 01:15:11.714491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.714458    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:15:11.716350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.716331    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4wnq"
Nov 02 01:15:11.720394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.720353    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:11.996637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.996608    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l"]
Nov 02 01:15:11.999684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:11.999654    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9fbg8"]
Nov 02 01:15:12.002852 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:15:12.002814    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2edcda9_40c1_4565_8843_1083f0f60b5a.slice/crio-0e0922a33117caf698ce1b0863df5b5240e35aebc7255d3bf31d491df0b672f6 WatchSource:0}: Error finding container 0e0922a33117caf698ce1b0863df5b5240e35aebc7255d3bf31d491df0b672f6: Status 404 returned error can't find the container with id 0e0922a33117caf698ce1b0863df5b5240e35aebc7255d3bf31d491df0b672f6
Nov 02 01:15:12.042482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.042109    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m4wnq"]
Nov 02 01:15:12.042482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.042176    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2hkrs"]
Nov 02 01:15:12.043190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.043174    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7zmf4"]
Nov 02 01:15:12.044328 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:15:12.044280    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be28aae_f92c_4eb1_b063_f84b46967f3b.slice/crio-3191ef635b26b1a5315cacf82b4bd96d0481bf36c7cbdec7d01fe5f36d5f07d1 WatchSource:0}: Error finding container 3191ef635b26b1a5315cacf82b4bd96d0481bf36c7cbdec7d01fe5f36d5f07d1: Status 404 returned error can't find the container with id 3191ef635b26b1a5315cacf82b4bd96d0481bf36c7cbdec7d01fe5f36d5f07d1
Nov 02 01:15:12.045356 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:15:12.045329    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7363cd_3783_4f09_b5de_ce703edde1d5.slice/crio-4db0afbdde2d0f5e0d5db70280f932c1eb49d2fc6f1eaf66b5a464e07b7a78be WatchSource:0}: Error finding container 4db0afbdde2d0f5e0d5db70280f932c1eb49d2fc6f1eaf66b5a464e07b7a78be: Status 404 returned error can't find the container with id 4db0afbdde2d0f5e0d5db70280f932c1eb49d2fc6f1eaf66b5a464e07b7a78be
Nov 02 01:15:12.046059 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:15:12.045994    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccf76ab_a4c9_4087_8ada_59b4c19218f8.slice/crio-d8073167555cd2cfbb78e11eff0b45f2a43b41aa572e0a48cf7dc4585e9e784c WatchSource:0}: Error finding container d8073167555cd2cfbb78e11eff0b45f2a43b41aa572e0a48cf7dc4585e9e784c: Status 404 returned error can't find the container with id d8073167555cd2cfbb78e11eff0b45f2a43b41aa572e0a48cf7dc4585e9e784c
Nov 02 01:15:12.151895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.151863    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76d8c5d8fd-g46xp"]
Nov 02 01:15:12.272563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.272530    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:12.404961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.404836    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:12.959350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.959311    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8" event={"ID":"b2edcda9-40c1-4565-8843-1083f0f60b5a","Type":"ContainerStarted","Data":"0e0922a33117caf698ce1b0863df5b5240e35aebc7255d3bf31d491df0b672f6"}
Nov 02 01:15:12.960593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.960565    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l" event={"ID":"91f2617a-9008-4f4a-ada4-8d5997064b52","Type":"ContainerStarted","Data":"a111727c9bc52be64ebfa07e7b00a2a6cdfbbd022cda99f0551370a51ff0dab1"}
Nov 02 01:15:12.962096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.962064    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hkrs" event={"ID":"0ccf76ab-a4c9-4087-8ada-59b4c19218f8","Type":"ContainerStarted","Data":"d8073167555cd2cfbb78e11eff0b45f2a43b41aa572e0a48cf7dc4585e9e784c"}
Nov 02 01:15:12.964195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.964160    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7zmf4" event={"ID":"1be28aae-f92c-4eb1-b063-f84b46967f3b","Type":"ContainerStarted","Data":"3191ef635b26b1a5315cacf82b4bd96d0481bf36c7cbdec7d01fe5f36d5f07d1"}
Nov 02 01:15:12.966157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.966123    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4wnq" event={"ID":"ab7363cd-3783-4f09-b5de-ce703edde1d5","Type":"ContainerStarted","Data":"5ae83eaa8d9e2f3ad140e65f5c8b5c598077cfaf42bc7d9af2c847dff27cddc0"}
Nov 02 01:15:12.966265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:12.966176    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4wnq" event={"ID":"ab7363cd-3783-4f09-b5de-ce703edde1d5","Type":"ContainerStarted","Data":"4db0afbdde2d0f5e0d5db70280f932c1eb49d2fc6f1eaf66b5a464e07b7a78be"}
Nov 02 01:15:13.973258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:13.973182    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hkrs" event={"ID":"0ccf76ab-a4c9-4087-8ada-59b4c19218f8","Type":"ContainerStarted","Data":"2393d4edabc79947a244620a14c06273b04436219835d2b64eda7b710c8e3634"}
Nov 02 01:15:13.973736 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:13.973279    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hkrs" event={"ID":"0ccf76ab-a4c9-4087-8ada-59b4c19218f8","Type":"ContainerStarted","Data":"08f08d5b0ed257bd1a7bfb08652f2c43eff5e372f15307811639ce00c2f3fdab"}
Nov 02 01:15:13.976509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:13.976478    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4wnq" event={"ID":"ab7363cd-3783-4f09-b5de-ce703edde1d5","Type":"ContainerStarted","Data":"a690480c1dca98663402b9580ac32810c2339c21720c8dfebc2f4e963a385002"}
Nov 02 01:15:13.979109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:13.979085    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9fbg8" event={"ID":"b2edcda9-40c1-4565-8843-1083f0f60b5a","Type":"ContainerStarted","Data":"5d0ac3e7d580a09c5678d507e1c561c4a3c84059ba99b961491c3940b3f29b51"}
Nov 02 01:15:14.817576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:14.817544    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:14.979654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:14.979569    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:14.983616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:14.983582    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l" event={"ID":"91f2617a-9008-4f4a-ada4-8d5997064b52","Type":"ContainerStarted","Data":"7210d1889ffbc7ae74845ced5f84913fbe481c0022cbb348a861ab549531194f"}
Nov 02 01:15:14.983777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:14.983643    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2v57l" event={"ID":"91f2617a-9008-4f4a-ada4-8d5997064b52","Type":"ContainerStarted","Data":"34ce1b8f84f4c903f60164f677be75e07743d83b48d3a7f0db3c2d6bdc5340bb"}
Nov 02 01:15:14.985276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:14.985256    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7zmf4" event={"ID":"1be28aae-f92c-4eb1-b063-f84b46967f3b","Type":"ContainerStarted","Data":"460f954a763fd14fb3e037afbb748be8f4b3ea4444f246f172cb057519a50212"}
Nov 02 01:15:14.985383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:14.985309    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7zmf4" event={"ID":"1be28aae-f92c-4eb1-b063-f84b46967f3b","Type":"ContainerStarted","Data":"5a746359faaf83520c40b8af3c0573d9b0d06eb5791e5971230e727f38de8631"}
Nov 02 01:15:14.985383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:14.985356    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:15:14.987391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:14.987368    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4wnq" event={"ID":"ab7363cd-3783-4f09-b5de-ce703edde1d5","Type":"ContainerStarted","Data":"d130390f86ff37d7b24631281ff44fcbc5a48dfff1961d036e757a4156c3b498"}
Nov 02 01:15:15.117661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:15.117613    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:15.246570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:15.246480    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:15.364689 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:15.364655    4851 patch_prober.go:29] interesting pod/image-registry-7978dd58c5-947pf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 01:15:15.364831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:15.364704    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7978dd58c5-947pf" podUID="d1b792a4-f821-4dec-9db4-36bfeceee2e6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:15:17.224839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:17.224808    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:17.597742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:17.597670    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:17.654698 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:17.654662    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:15:19.155601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.155561    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:19.300176 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.300146    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:19.417370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.417338    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wz5wp"]
Nov 02 01:15:19.422664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.422647    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.459578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.459548    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:19.472384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.472352    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pc7pp\""
Nov 02 01:15:19.472612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.472587    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 01:15:19.472612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.472594    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 01:15:19.489013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.488987    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 01:15:19.489118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.489027    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 01:15:19.554348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.554318    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/deecdad1-584c-40c7-827b-17738a003035-node-exporter-tls\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.554478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.554354    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/deecdad1-584c-40c7-827b-17738a003035-root\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.554478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.554379    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deecdad1-584c-40c7-827b-17738a003035-metrics-client-ca\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.554478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.554440    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/deecdad1-584c-40c7-827b-17738a003035-node-exporter-accelerators-collector-config\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.554598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.554528    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/deecdad1-584c-40c7-827b-17738a003035-node-exporter-textfile\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.554598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.554562    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deecdad1-584c-40c7-827b-17738a003035-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.554659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.554632    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deecdad1-584c-40c7-827b-17738a003035-sys\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.554742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.554660    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bq6\" (UniqueName: \"kubernetes.io/projected/deecdad1-584c-40c7-827b-17738a003035-kube-api-access-k8bq6\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.554742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.554682    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/deecdad1-584c-40c7-827b-17738a003035-node-exporter-wtmp\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.587817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.587786    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:19.655208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655176    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bq6\" (UniqueName: \"kubernetes.io/projected/deecdad1-584c-40c7-827b-17738a003035-kube-api-access-k8bq6\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655212    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/deecdad1-584c-40c7-827b-17738a003035-node-exporter-wtmp\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655242    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/deecdad1-584c-40c7-827b-17738a003035-node-exporter-tls\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655261    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/deecdad1-584c-40c7-827b-17738a003035-root\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655314    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deecdad1-584c-40c7-827b-17738a003035-metrics-client-ca\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655342    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/deecdad1-584c-40c7-827b-17738a003035-node-exporter-accelerators-collector-config\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655378    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/deecdad1-584c-40c7-827b-17738a003035-node-exporter-textfile\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655407    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/deecdad1-584c-40c7-827b-17738a003035-node-exporter-wtmp\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655421    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/deecdad1-584c-40c7-827b-17738a003035-root\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655411    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deecdad1-584c-40c7-827b-17738a003035-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655501    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deecdad1-584c-40c7-827b-17738a003035-sys\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655581    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deecdad1-584c-40c7-827b-17738a003035-sys\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.655934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.655738    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/deecdad1-584c-40c7-827b-17738a003035-node-exporter-textfile\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.656149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.656060    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deecdad1-584c-40c7-827b-17738a003035-metrics-client-ca\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.656149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.656070    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/deecdad1-584c-40c7-827b-17738a003035-node-exporter-accelerators-collector-config\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.657705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.657687    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/deecdad1-584c-40c7-827b-17738a003035-node-exporter-tls\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.657778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.657731    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deecdad1-584c-40c7-827b-17738a003035-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.667618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.667564    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bq6\" (UniqueName: \"kubernetes.io/projected/deecdad1-584c-40c7-827b-17738a003035-kube-api-access-k8bq6\") pod \"node-exporter-wz5wp\" (UID: \"deecdad1-584c-40c7-827b-17738a003035\") " pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.707132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.707093    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:19.740090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.740070    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wz5wp"
Nov 02 01:15:19.753333 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:15:19.753308    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeecdad1_584c_40c7_827b_17738a003035.slice/crio-d5ab555155d8be5b30befa3cbabfc2b356d24ab3099c876b873b9fc74361feda WatchSource:0}: Error finding container d5ab555155d8be5b30befa3cbabfc2b356d24ab3099c876b873b9fc74361feda: Status 404 returned error can't find the container with id d5ab555155d8be5b30befa3cbabfc2b356d24ab3099c876b873b9fc74361feda
Nov 02 01:15:19.840958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.840927    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:19.979766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:19.979704    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:20.007552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:20.007529    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wz5wp" event={"ID":"deecdad1-584c-40c7-827b-17738a003035","Type":"ContainerStarted","Data":"d5ab555155d8be5b30befa3cbabfc2b356d24ab3099c876b873b9fc74361feda"}
Nov 02 01:15:20.422199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:20.422126    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:20.618894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:20.618859    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:22.018330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:22.018277    4851 generic.go:356] "Generic (PLEG): container finished" podID="deecdad1-584c-40c7-827b-17738a003035" containerID="4e6681f0759ad5748370ddbef6090a2b0f8680159ab9529abd65a21667f120a2" exitCode=0
Nov 02 01:15:22.018740 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:22.018365    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wz5wp" event={"ID":"deecdad1-584c-40c7-827b-17738a003035","Type":"ContainerDied","Data":"4e6681f0759ad5748370ddbef6090a2b0f8680159ab9529abd65a21667f120a2"}
Nov 02 01:15:22.157463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:22.157440    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:15:22.365726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:22.365644    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:22.567593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:22.567566    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:22.696275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:22.696239    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:22.817259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:22.817228    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:23.023901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:23.023828    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wz5wp" event={"ID":"deecdad1-584c-40c7-827b-17738a003035","Type":"ContainerStarted","Data":"3714826b2be56f015d8eed5c86124f87109a294abac98f3bbdce33e060006632"}
Nov 02 01:15:23.023901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:23.023875    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wz5wp" event={"ID":"deecdad1-584c-40c7-827b-17738a003035","Type":"ContainerStarted","Data":"1a30a9b72bf779e44f60b49f253f2029665a3343a3ceeb620e5ff3b2d1d1daa2"}
Nov 02 01:15:23.046172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:23.046119    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wz5wp" podStartSLOduration=2.583872265 podStartE2EDuration="4.046106531s" totalImagesPullingTime="1.388820169s" totalInitContainerRuntime="73.414097ms" isStatefulPod=true podCreationTimestamp="2026-11-02 01:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:23.045257839 +0000 UTC m=+28.350190666" watchObservedRunningTime="2026-11-02 01:15:23.046106531 +0000 UTC m=+28.351039351"
Nov 02 01:15:23.703268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:23.703235    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:15:23.931576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:23.931543    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:24.056381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:24.056278    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:24.197139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:24.197113    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:24.327329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:24.327231    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:24.477489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:24.477462    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:24.905234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:24.905207    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tkzsw"
Nov 02 01:15:24.994040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:24.994008    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7zmf4"
Nov 02 01:15:25.671481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:25.671449    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:26.132099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:26.132028    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:26.259178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:26.259146    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:31.191629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:31.191593    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:31.313312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:31.313243    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:31.803435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:31.803396    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:31.934639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:31.934607    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:34.681197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:34.681168    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7978dd58c5-947pf"]
Nov 02 01:15:34.850556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:34.850529    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:34.977953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:34.977866    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:35.809034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:35.808992    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:35.935638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:35.935599    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:37.172917 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.172873    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp" podUID="fb996d25-e713-4ed5-a1db-871234d5a8cd" containerName="registry" containerID="cri-o://035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f" gracePeriod=30
Nov 02 01:15:37.406732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.406710    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:15:37.487257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487192    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-certificates\") pod \"fb996d25-e713-4ed5-a1db-871234d5a8cd\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") "
Nov 02 01:15:37.487257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487235    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-installation-pull-secrets\") pod \"fb996d25-e713-4ed5-a1db-871234d5a8cd\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") "
Nov 02 01:15:37.487498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487267    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-bound-sa-token\") pod \"fb996d25-e713-4ed5-a1db-871234d5a8cd\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") "
Nov 02 01:15:37.487498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487319    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-image-registry-private-configuration\") pod \"fb996d25-e713-4ed5-a1db-871234d5a8cd\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") "
Nov 02 01:15:37.487498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487352    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-trusted-ca\") pod \"fb996d25-e713-4ed5-a1db-871234d5a8cd\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") "
Nov 02 01:15:37.487498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487379    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") pod \"fb996d25-e713-4ed5-a1db-871234d5a8cd\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") "
Nov 02 01:15:37.487498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487412    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-kube-api-access-bn6dd\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-kube-api-access-bn6dd\") pod \"fb996d25-e713-4ed5-a1db-871234d5a8cd\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") "
Nov 02 01:15:37.487498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487458    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb996d25-e713-4ed5-a1db-871234d5a8cd-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb996d25-e713-4ed5-a1db-871234d5a8cd-ca-trust-extracted\") pod \"fb996d25-e713-4ed5-a1db-871234d5a8cd\" (UID: \"fb996d25-e713-4ed5-a1db-871234d5a8cd\") "
Nov 02 01:15:37.487807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487590    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-certificates" pod "fb996d25-e713-4ed5-a1db-871234d5a8cd" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:15:37.487807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487785    4851 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-certificates\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:15:37.487922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.487840    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-trusted-ca" pod "fb996d25-e713-4ed5-a1db-871234d5a8cd" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:15:37.490077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.490019    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-bound-sa-token" pod "fb996d25-e713-4ed5-a1db-871234d5a8cd" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:15:37.490207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.490100    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-image-registry-private-configuration" pod "fb996d25-e713-4ed5-a1db-871234d5a8cd" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:15:37.490207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.490103    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls" pod "fb996d25-e713-4ed5-a1db-871234d5a8cd" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:15:37.490352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.490221    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-installation-pull-secrets" pod "fb996d25-e713-4ed5-a1db-871234d5a8cd" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:15:37.490455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.490433    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-kube-api-access-bn6dd" pod "fb996d25-e713-4ed5-a1db-871234d5a8cd" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd"). InnerVolumeSpecName "kube-api-access-bn6dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:15:37.494586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.494565    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb996d25-e713-4ed5-a1db-871234d5a8cd-ca-trust-extracted" pod "fb996d25-e713-4ed5-a1db-871234d5a8cd" (UID: "fb996d25-e713-4ed5-a1db-871234d5a8cd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:15:37.588525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.588498    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn6dd\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-kube-api-access-bn6dd\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:15:37.588525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.588521    4851 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb996d25-e713-4ed5-a1db-871234d5a8cd-ca-trust-extracted\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:15:37.588670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.588531    4851 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-installation-pull-secrets\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:15:37.588670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.588542    4851 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-bound-sa-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:15:37.588670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.588552    4851 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb996d25-e713-4ed5-a1db-871234d5a8cd-image-registry-private-configuration\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:15:37.588670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.588561    4851 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb996d25-e713-4ed5-a1db-871234d5a8cd-trusted-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:15:37.588670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.588570    4851 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb996d25-e713-4ed5-a1db-871234d5a8cd-registry-tls\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:15:37.813546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.813462    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:37.946870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:37.946836    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:38.081075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.081000    4851 generic.go:356] "Generic (PLEG): container finished" podID="fb996d25-e713-4ed5-a1db-871234d5a8cd" containerID="035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f" exitCode=0
Nov 02 01:15:38.081075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.081054    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp"
Nov 02 01:15:38.081269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.081082    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp" event={"ID":"fb996d25-e713-4ed5-a1db-871234d5a8cd","Type":"ContainerDied","Data":"035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f"}
Nov 02 01:15:38.081269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.081135    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76d8c5d8fd-g46xp" event={"ID":"fb996d25-e713-4ed5-a1db-871234d5a8cd","Type":"ContainerDied","Data":"4b2123c0e16336065bbba150c11fc2182b294dd71bedb906b4d208328d374eb5"}
Nov 02 01:15:38.081269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.081155    4851 scope.go:122] "RemoveContainer" containerID="035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f"
Nov 02 01:15:38.090881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.090865    4851 scope.go:122] "RemoveContainer" containerID="035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f"
Nov 02 01:15:38.091177 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:38.091152    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f\": container with ID starting with 035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f not found: ID does not exist" containerID="035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f"
Nov 02 01:15:38.091222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.091185    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f"} err="failed to get container status \"035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f\": rpc error: code = NotFound desc = could not find container \"035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f\": container with ID starting with 035a34465728e9f17ef8e32afaf584a957d0a98e587d46a6f6669ed3685c7c8f not found: ID does not exist"
Nov 02 01:15:38.103779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.103756    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76d8c5d8fd-g46xp"]
Nov 02 01:15:38.108907 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.108887    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76d8c5d8fd-g46xp"]
Nov 02 01:15:38.121179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.121155    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:38.259608 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.259582    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:38.843724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:38.843688    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb996d25-e713-4ed5-a1db-871234d5a8cd" path="/var/lib/kubelet/pods/fb996d25-e713-4ed5-a1db-871234d5a8cd/volumes"
Nov 02 01:15:39.412606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:39.412554    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:39.541485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:39.541445    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:41.107517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:41.107467    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:41.235425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:41.235394    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:43.618205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:43.618172    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:43.786106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:43.786073    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:44.899688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:44.899652    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:45.066985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:45.066954    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:45.815063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:45.815029    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:46.014521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:46.014489    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:47.278540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:47.278506    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:47.414460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:47.414431    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:47.655180 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:15:47.655141    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:15:56.114462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:56.114430    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:56.242490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:56.242459    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:15:59.701316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:59.701251    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7978dd58c5-947pf" podUID="d1b792a4-f821-4dec-9db4-36bfeceee2e6" containerName="registry" containerID="cri-o://42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b" gracePeriod=30
Nov 02 01:15:59.947844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:15:59.947823    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:16:00.066566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.066492    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-kube-api-access-bfpxv\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-kube-api-access-bfpxv\") pod \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") "
Nov 02 01:16:00.066566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.066540    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-bound-sa-token\") pod \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") "
Nov 02 01:16:00.066788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.066580    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-image-registry-private-configuration\") pod \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") "
Nov 02 01:16:00.066788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.066604    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b792a4-f821-4dec-9db4-36bfeceee2e6-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1b792a4-f821-4dec-9db4-36bfeceee2e6-ca-trust-extracted\") pod \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") "
Nov 02 01:16:00.066788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.066643    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-certificates\") pod \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") "
Nov 02 01:16:00.066788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.066669    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-installation-pull-secrets\") pod \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") "
Nov 02 01:16:00.066788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.066755    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-trusted-ca\") pod \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") "
Nov 02 01:16:00.067038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.066834    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") pod \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\" (UID: \"d1b792a4-f821-4dec-9db4-36bfeceee2e6\") "
Nov 02 01:16:00.067171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.067142    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-certificates" pod "d1b792a4-f821-4dec-9db4-36bfeceee2e6" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:16:00.067841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.067812    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-trusted-ca" pod "d1b792a4-f821-4dec-9db4-36bfeceee2e6" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:16:00.069222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.069199    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-installation-pull-secrets" pod "d1b792a4-f821-4dec-9db4-36bfeceee2e6" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:16:00.069599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.069557    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-kube-api-access-bfpxv" pod "d1b792a4-f821-4dec-9db4-36bfeceee2e6" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6"). InnerVolumeSpecName "kube-api-access-bfpxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:16:00.069683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.069637    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-bound-sa-token" pod "d1b792a4-f821-4dec-9db4-36bfeceee2e6" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:16:00.069683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.069666    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls" pod "d1b792a4-f821-4dec-9db4-36bfeceee2e6" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:16:00.069758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.069700    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-image-registry-private-configuration" pod "d1b792a4-f821-4dec-9db4-36bfeceee2e6" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:16:00.075440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.075413    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b792a4-f821-4dec-9db4-36bfeceee2e6-ca-trust-extracted" pod "d1b792a4-f821-4dec-9db4-36bfeceee2e6" (UID: "d1b792a4-f821-4dec-9db4-36bfeceee2e6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:16:00.168032 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.168000    4851 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-installation-pull-secrets\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:16:00.168152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.168036    4851 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-trusted-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:16:00.168152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.168053    4851 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-tls\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:16:00.168152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.168067    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfpxv\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-kube-api-access-bfpxv\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:16:00.168152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.168081    4851 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1b792a4-f821-4dec-9db4-36bfeceee2e6-bound-sa-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:16:00.168152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.168096    4851 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1b792a4-f821-4dec-9db4-36bfeceee2e6-image-registry-private-configuration\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:16:00.168152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.168110    4851 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1b792a4-f821-4dec-9db4-36bfeceee2e6-ca-trust-extracted\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:16:00.168152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.168124    4851 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1b792a4-f821-4dec-9db4-36bfeceee2e6-registry-certificates\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:16:00.171777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.171747    4851 generic.go:356] "Generic (PLEG): container finished" podID="d1b792a4-f821-4dec-9db4-36bfeceee2e6" containerID="42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b" exitCode=0
Nov 02 01:16:00.171891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.171796    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7978dd58c5-947pf"
Nov 02 01:16:00.171891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.171831    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7978dd58c5-947pf" event={"ID":"d1b792a4-f821-4dec-9db4-36bfeceee2e6","Type":"ContainerDied","Data":"42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b"}
Nov 02 01:16:00.171891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.171879    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7978dd58c5-947pf" event={"ID":"d1b792a4-f821-4dec-9db4-36bfeceee2e6","Type":"ContainerDied","Data":"9acc222948a3d1cea2f0edf0bd794d8d4b8889f082defba66de4e258cdb72602"}
Nov 02 01:16:00.172013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.171898    4851 scope.go:122] "RemoveContainer" containerID="42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b"
Nov 02 01:16:00.180079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.180064    4851 scope.go:122] "RemoveContainer" containerID="42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b"
Nov 02 01:16:00.180368 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:16:00.180347    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b\": container with ID starting with 42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b not found: ID does not exist" containerID="42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b"
Nov 02 01:16:00.180416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.180376    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b"} err="failed to get container status \"42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b\": rpc error: code = NotFound desc = could not find container \"42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b\": container with ID starting with 42c25a50af06e2b2e3cd0686618088e0a66dbe264fb2f73fddee8f9dbca50b3b not found: ID does not exist"
Nov 02 01:16:00.193533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.193514    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7978dd58c5-947pf"]
Nov 02 01:16:00.197560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.197540    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7978dd58c5-947pf"]
Nov 02 01:16:00.844048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:00.844020    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b792a4-f821-4dec-9db4-36bfeceee2e6" path="/var/lib/kubelet/pods/d1b792a4-f821-4dec-9db4-36bfeceee2e6/volumes"
Nov 02 01:16:04.954505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:04.954463    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:05.086225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:05.086195    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:06.858912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:06.858881    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:06.983451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:06.983420    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:08.344125 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:08.344086    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:08.488888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:08.488854    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:08.619263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:08.619185    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:10.331638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:10.331599    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:10.459772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:10.459741    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:11.150154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:11.150123    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:11.211862 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:11.211830    4851 generic.go:356] "Generic (PLEG): container finished" podID="cc662357-7bcd-4efb-9857-37a197cb67bf" containerID="de01b9034a2e7e30c34f82c93752d0bb4741e83db51417c2152f2775f81adfba" exitCode=0
Nov 02 01:16:11.212024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:11.211903    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr" event={"ID":"cc662357-7bcd-4efb-9857-37a197cb67bf","Type":"ContainerDied","Data":"de01b9034a2e7e30c34f82c93752d0bb4741e83db51417c2152f2775f81adfba"}
Nov 02 01:16:11.212236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:11.212209    4851 scope.go:122] "RemoveContainer" containerID="de01b9034a2e7e30c34f82c93752d0bb4741e83db51417c2152f2775f81adfba"
Nov 02 01:16:11.275997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:11.275944    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:12.217628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:12.217597    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-v47hr" event={"ID":"cc662357-7bcd-4efb-9857-37a197cb67bf","Type":"ContainerStarted","Data":"cd4dee6fa79d2105bd2e960d0da703bb5802064c87bb3f5cd92e755cd7bce21c"}
Nov 02 01:16:16.233713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.233678    4851 generic.go:356] "Generic (PLEG): container finished" podID="16b92078-b559-44f6-85b0-4f9d1da9ddcf" containerID="c4fe19debf008433ac40c1c2e73686ae07e02f906a063c677031773da05224b3" exitCode=0
Nov 02 01:16:16.234126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.233751    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bdwzr" event={"ID":"16b92078-b559-44f6-85b0-4f9d1da9ddcf","Type":"ContainerDied","Data":"c4fe19debf008433ac40c1c2e73686ae07e02f906a063c677031773da05224b3"}
Nov 02 01:16:16.234126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.234092    4851 scope.go:122] "RemoveContainer" containerID="c4fe19debf008433ac40c1c2e73686ae07e02f906a063c677031773da05224b3"
Nov 02 01:16:16.795182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.795149    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-dlztf-6fpck"]
Nov 02 01:16:16.795481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.795469    4851 cpu_manager.go:414] "containerMap: removing container" podUID="fb996d25-e713-4ed5-a1db-871234d5a8cd" containerName="registry"
Nov 02 01:16:16.795534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.795483    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb996d25-e713-4ed5-a1db-871234d5a8cd" containerName="registry"
Nov 02 01:16:16.795534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.795497    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d1b792a4-f821-4dec-9db4-36bfeceee2e6" containerName="registry"
Nov 02 01:16:16.795534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.795502    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b792a4-f821-4dec-9db4-36bfeceee2e6" containerName="registry"
Nov 02 01:16:16.795629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.795550    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b792a4-f821-4dec-9db4-36bfeceee2e6" containerName="registry"
Nov 02 01:16:16.795629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.795562    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb996d25-e713-4ed5-a1db-871234d5a8cd" containerName="registry"
Nov 02 01:16:16.798419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.798404    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.808989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.808963    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dlztf-6fpck"]
Nov 02 01:16:16.895604 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.895572    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjhxf\" (UniqueName: \"kubernetes.io/projected/52de128e-db62-4aca-87e6-2af6c1c90a26-kube-api-access-gjhxf\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.895763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.895623    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-trusted-ca-bundle\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.895763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.895724    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-service-ca-bundle\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.895834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.895771    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/52de128e-db62-4aca-87e6-2af6c1c90a26-archives-path\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.996074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.996044    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjhxf\" (UniqueName: \"kubernetes.io/projected/52de128e-db62-4aca-87e6-2af6c1c90a26-kube-api-access-gjhxf\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.996228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.996092    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-trusted-ca-bundle\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.996272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.996220    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-service-ca-bundle\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.996272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.996257    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/52de128e-db62-4aca-87e6-2af6c1c90a26-archives-path\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.996643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.996622    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/52de128e-db62-4aca-87e6-2af6c1c90a26-archives-path\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.996857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.996840    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-service-ca-bundle\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:16.996914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:16.996899    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-trusted-ca-bundle\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:17.008582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:17.008550    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjhxf\" (UniqueName: \"kubernetes.io/projected/52de128e-db62-4aca-87e6-2af6c1c90a26-kube-api-access-gjhxf\") pod \"periodic-gathering-dlztf-6fpck\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") " pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:17.109582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:17.109507    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:16:17.239309 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:17.239250    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bdwzr" event={"ID":"16b92078-b559-44f6-85b0-4f9d1da9ddcf","Type":"ContainerStarted","Data":"9a7a58a468def2181348298de3cba8455bf7864afd3054a877a5e5c885895138"}
Nov 02 01:16:17.379477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:17.379411    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dlztf-6fpck"]
Nov 02 01:16:17.382497 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:16:17.382467    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52de128e_db62_4aca_87e6_2af6c1c90a26.slice/crio-a69fad20d20e8b0a808594cfa3783557ce0d5989821801d998819e3f37e0b95f WatchSource:0}: Error finding container a69fad20d20e8b0a808594cfa3783557ce0d5989821801d998819e3f37e0b95f: Status 404 returned error can't find the container with id a69fad20d20e8b0a808594cfa3783557ce0d5989821801d998819e3f37e0b95f
Nov 02 01:16:17.603995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:17.603962    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:17.654340 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:16:17.654239    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:16:17.734488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:17.734462    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:18.244511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:18.244479    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dlztf-6fpck" event={"ID":"52de128e-db62-4aca-87e6-2af6c1c90a26","Type":"ContainerStarted","Data":"1adc2953bb3c045ed6938525cccacf56c71057dcd75c7a26d1c93e2b498d946c"}
Nov 02 01:16:18.244906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:18.244530    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dlztf-6fpck" event={"ID":"52de128e-db62-4aca-87e6-2af6c1c90a26","Type":"ContainerStarted","Data":"a69fad20d20e8b0a808594cfa3783557ce0d5989821801d998819e3f37e0b95f"}
Nov 02 01:16:18.262497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:18.262458    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-dlztf-6fpck" podStartSLOduration=2.2624454800000002 podStartE2EDuration="2.26244548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:16:18.261859851 +0000 UTC m=+83.566792678" watchObservedRunningTime="2026-11-02 01:16:18.26244548 +0000 UTC m=+83.567378306"
Nov 02 01:16:26.107566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:26.107518    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:26.237477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:26.237441    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:26.275917 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:26.275888    4851 generic.go:356] "Generic (PLEG): container finished" podID="414ef228-1302-4af5-8118-be23a4280494" containerID="58e5686e77211bdee714847c4d2a7f0b0041138bd6cc91e55964995d3291aa9e" exitCode=0
Nov 02 01:16:26.276064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:26.275962    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw" event={"ID":"414ef228-1302-4af5-8118-be23a4280494","Type":"ContainerDied","Data":"58e5686e77211bdee714847c4d2a7f0b0041138bd6cc91e55964995d3291aa9e"}
Nov 02 01:16:26.276273 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:26.276260    4851 scope.go:122] "RemoveContainer" containerID="58e5686e77211bdee714847c4d2a7f0b0041138bd6cc91e55964995d3291aa9e"
Nov 02 01:16:27.282425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:27.282389    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-g6jbw" event={"ID":"414ef228-1302-4af5-8118-be23a4280494","Type":"ContainerStarted","Data":"a5cbaecdedae84f4b74cb01b97ab8b6a892cdc04d1e2d5e49242caf25e427462"}
Nov 02 01:16:34.902622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:34.902588    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:44.965685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:16:44.965651    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:16:47.653956 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:16:47.653907    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:17:17.654168 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:17:17.654086    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:17:21.811421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:21.811393    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:17:40.547471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:40.547387    4851 generic.go:356] "Generic (PLEG): container finished" podID="52de128e-db62-4aca-87e6-2af6c1c90a26" containerID="1adc2953bb3c045ed6938525cccacf56c71057dcd75c7a26d1c93e2b498d946c" exitCode=0
Nov 02 01:17:40.547471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:40.547423    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dlztf-6fpck" event={"ID":"52de128e-db62-4aca-87e6-2af6c1c90a26","Type":"ContainerDied","Data":"1adc2953bb3c045ed6938525cccacf56c71057dcd75c7a26d1c93e2b498d946c"}
Nov 02 01:17:41.696454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.696431    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:17:41.846820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.846734    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-trusted-ca-bundle\") pod \"52de128e-db62-4aca-87e6-2af6c1c90a26\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") "
Nov 02 01:17:41.846820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.846809    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de128e-db62-4aca-87e6-2af6c1c90a26-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/52de128e-db62-4aca-87e6-2af6c1c90a26-archives-path\") pod \"52de128e-db62-4aca-87e6-2af6c1c90a26\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") "
Nov 02 01:17:41.847003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.846969    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-service-ca-bundle\") pod \"52de128e-db62-4aca-87e6-2af6c1c90a26\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") "
Nov 02 01:17:41.847053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.847018    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52de128e-db62-4aca-87e6-2af6c1c90a26-kube-api-access-gjhxf\" (UniqueName: \"kubernetes.io/projected/52de128e-db62-4aca-87e6-2af6c1c90a26-kube-api-access-gjhxf\") pod \"52de128e-db62-4aca-87e6-2af6c1c90a26\" (UID: \"52de128e-db62-4aca-87e6-2af6c1c90a26\") "
Nov 02 01:17:41.847099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.847073    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de128e-db62-4aca-87e6-2af6c1c90a26-archives-path" pod "52de128e-db62-4aca-87e6-2af6c1c90a26" (UID: "52de128e-db62-4aca-87e6-2af6c1c90a26"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:17:41.847244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.847231    4851 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/52de128e-db62-4aca-87e6-2af6c1c90a26-archives-path\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:17:41.847280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.847149    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-trusted-ca-bundle" pod "52de128e-db62-4aca-87e6-2af6c1c90a26" (UID: "52de128e-db62-4aca-87e6-2af6c1c90a26"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:17:41.847354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.847339    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-service-ca-bundle" pod "52de128e-db62-4aca-87e6-2af6c1c90a26" (UID: "52de128e-db62-4aca-87e6-2af6c1c90a26"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:17:41.849283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.849266    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52de128e-db62-4aca-87e6-2af6c1c90a26-kube-api-access-gjhxf" pod "52de128e-db62-4aca-87e6-2af6c1c90a26" (UID: "52de128e-db62-4aca-87e6-2af6c1c90a26"). InnerVolumeSpecName "kube-api-access-gjhxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:17:41.947976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.947934    4851 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-service-ca-bundle\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:17:41.947976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.947970    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjhxf\" (UniqueName: \"kubernetes.io/projected/52de128e-db62-4aca-87e6-2af6c1c90a26-kube-api-access-gjhxf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:17:41.947976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:41.947986    4851 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52de128e-db62-4aca-87e6-2af6c1c90a26-trusted-ca-bundle\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:17:42.555947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:42.555902    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dlztf-6fpck" event={"ID":"52de128e-db62-4aca-87e6-2af6c1c90a26","Type":"ContainerDied","Data":"a69fad20d20e8b0a808594cfa3783557ce0d5989821801d998819e3f37e0b95f"}
Nov 02 01:17:42.555947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:42.555920    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dlztf-6fpck"
Nov 02 01:17:42.556167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:42.555986    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a69fad20d20e8b0a808594cfa3783557ce0d5989821801d998819e3f37e0b95f"
Nov 02 01:17:43.086863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:43.086830    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:17:43.216462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:43.216429    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:17:45.023714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:45.023680    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:17:47.653934 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:17:47.653899    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:17:49.106479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:49.106444    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:17:49.231598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:17:49.231569    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:05.056830 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:05.056778    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:16.866832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:16.866797    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:17.654869 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:18:17.654831    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:18:44.984476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:44.984389    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:45.165460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.165422    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:45.879418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.879383    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"]
Nov 02 01:18:45.879736 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.879719    4851 cpu_manager.go:414] "containerMap: removing container" podUID="52de128e-db62-4aca-87e6-2af6c1c90a26" containerName="insights-gathering"
Nov 02 01:18:45.879818 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.879738    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52de128e-db62-4aca-87e6-2af6c1c90a26" containerName="insights-gathering"
Nov 02 01:18:45.879880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.879858    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="52de128e-db62-4aca-87e6-2af6c1c90a26" containerName="insights-gathering"
Nov 02 01:18:45.882923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.882901    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:45.890308 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.890275    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 01:18:45.891552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.891533    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 01:18:45.891646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.891627    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-nh7nl\""
Nov 02 01:18:45.894513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.894492    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"]
Nov 02 01:18:45.988387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.988355    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:45.988387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.988388    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbhx\" (UniqueName: \"kubernetes.io/projected/0fe84d39-b955-45e2-84ad-f4384d17e705-kube-api-access-wsbhx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:45.988797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:45.988433    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:46.089496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.089465    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:46.089682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.089550    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:46.089682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.089579    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbhx\" (UniqueName: \"kubernetes.io/projected/0fe84d39-b955-45e2-84ad-f4384d17e705-kube-api-access-wsbhx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:46.089937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.089913    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:46.089997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.089924    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:46.104904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.104880    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbhx\" (UniqueName: \"kubernetes.io/projected/0fe84d39-b955-45e2-84ad-f4384d17e705-kube-api-access-wsbhx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:46.198966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.198909    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:18:46.482329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.482125    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"]
Nov 02 01:18:46.485172 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:18:46.485140    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe84d39_b955_45e2_84ad_f4384d17e705.slice/crio-174e3afa749363ee3d4ef48164a364212e66d1b81a4f6b2161e3e9970f3e12b9 WatchSource:0}: Error finding container 174e3afa749363ee3d4ef48164a364212e66d1b81a4f6b2161e3e9970f3e12b9: Status 404 returned error can't find the container with id 174e3afa749363ee3d4ef48164a364212e66d1b81a4f6b2161e3e9970f3e12b9
Nov 02 01:18:46.497244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.497221    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"]
Nov 02 01:18:46.501381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.501366    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.505373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.505350    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 01:18:46.511059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.511031    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tdsjr\""
Nov 02 01:18:46.511059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.511054    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 01:18:46.511190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.511058    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 01:18:46.511190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.511057    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 01:18:46.514257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.514234    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"]
Nov 02 01:18:46.554807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.554766    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:46.594569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.594545    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.594673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.594573    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.594673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.594594    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.594673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.594610    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.594770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.594738    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.594862 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.594771    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4mw\" (UniqueName: \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-kube-api-access-zt4mw\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.594862 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.594806    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.678923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.678893    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:46.696014 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.695987    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.696177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.696027    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.696177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.696050    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4mw\" (UniqueName: \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-kube-api-access-zt4mw\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.696177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.696071    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.696177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.696126    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.696177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.696145    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.696177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.696172    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.696786 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.696754    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.698639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.698615    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.698746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.698734    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.698960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.698941    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.699078 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.699062    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.708591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.708570    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4mw\" (UniqueName: \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-kube-api-access-zt4mw\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.708591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.708588    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2c629\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:46.777937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.777881    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx" event={"ID":"0fe84d39-b955-45e2-84ad-f4384d17e705","Type":"ContainerStarted","Data":"174e3afa749363ee3d4ef48164a364212e66d1b81a4f6b2161e3e9970f3e12b9"}
Nov 02 01:18:46.813311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:46.813269    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:47.089329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:47.089244    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"]
Nov 02 01:18:47.092751 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:18:47.092722    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1fa953_1b1b_4590_921f_68d7fa99e9a9.slice/crio-eb896e463cb7bcf663e669805405d20f0b6e6178c54917bd806509bb5e198001 WatchSource:0}: Error finding container eb896e463cb7bcf663e669805405d20f0b6e6178c54917bd806509bb5e198001: Status 404 returned error can't find the container with id eb896e463cb7bcf663e669805405d20f0b6e6178c54917bd806509bb5e198001
Nov 02 01:18:47.657015 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:18:47.656984    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:18:47.783127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:47.783096    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629" event={"ID":"3b1fa953-1b1b-4590-921f-68d7fa99e9a9","Type":"ContainerStarted","Data":"eb896e463cb7bcf663e669805405d20f0b6e6178c54917bd806509bb5e198001"}
Nov 02 01:18:47.855138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:47.855113    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:51.279993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:51.279958    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:18:51.280534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:51.280024    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:18:51.804855 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:51.804817    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629" event={"ID":"3b1fa953-1b1b-4590-921f-68d7fa99e9a9","Type":"ContainerStarted","Data":"4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7"}
Nov 02 01:18:51.805060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:51.804973    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:51.831639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:51.831581    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629" podStartSLOduration=1.64764841 podStartE2EDuration="5.831562423s" totalImagesPullingTime="4.183914013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:18:51.829115181 +0000 UTC m=+237.134048018" watchObservedRunningTime="2026-11-02 01:18:51.831562423 +0000 UTC m=+237.136495249"
Nov 02 01:18:51.864302 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:51.864262    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:51.996098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:51.996060    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:52.811713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:52.811489    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:18:52.907077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:52.907014    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:53.036190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:53.036156    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:53.162908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:53.162877    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:18:54.819515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:54.819479    4851 generic.go:356] "Generic (PLEG): container finished" podID="0fe84d39-b955-45e2-84ad-f4384d17e705" containerID="be424356165225d8ef03857f653cf1d94420c6b3361e125a3a4a1a0e19e79092" exitCode=0
Nov 02 01:18:54.819989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:54.819622    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx" event={"ID":"0fe84d39-b955-45e2-84ad-f4384d17e705","Type":"ContainerDied","Data":"be424356165225d8ef03857f653cf1d94420c6b3361e125a3a4a1a0e19e79092"}
Nov 02 01:18:55.435480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:18:55.435445    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:01.850508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:01.850471    4851 generic.go:356] "Generic (PLEG): container finished" podID="0fe84d39-b955-45e2-84ad-f4384d17e705" containerID="914cce7781f50b429d6d69578dd0a4e160281aa14640934019c054161d172e35" exitCode=0
Nov 02 01:19:01.851089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:01.850525    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx" event={"ID":"0fe84d39-b955-45e2-84ad-f4384d17e705","Type":"ContainerDied","Data":"914cce7781f50b429d6d69578dd0a4e160281aa14640934019c054161d172e35"}
Nov 02 01:19:10.889121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:10.889039    4851 generic.go:356] "Generic (PLEG): container finished" podID="0fe84d39-b955-45e2-84ad-f4384d17e705" containerID="6d7a43be9889141ddd473ecbf38448bf3ac18aa58e410a2a80bea9e8c4d4eeaf" exitCode=0
Nov 02 01:19:10.889121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:10.889103    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx" event={"ID":"0fe84d39-b955-45e2-84ad-f4384d17e705","Type":"ContainerDied","Data":"6d7a43be9889141ddd473ecbf38448bf3ac18aa58e410a2a80bea9e8c4d4eeaf"}
Nov 02 01:19:12.016034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.016010    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:19:12.112667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.112631    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-util\" (UniqueName: \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-util\") pod \"0fe84d39-b955-45e2-84ad-f4384d17e705\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") "
Nov 02 01:19:12.112667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.112674    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe84d39-b955-45e2-84ad-f4384d17e705-kube-api-access-wsbhx\" (UniqueName: \"kubernetes.io/projected/0fe84d39-b955-45e2-84ad-f4384d17e705-kube-api-access-wsbhx\") pod \"0fe84d39-b955-45e2-84ad-f4384d17e705\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") "
Nov 02 01:19:12.112920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.112710    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-bundle\") pod \"0fe84d39-b955-45e2-84ad-f4384d17e705\" (UID: \"0fe84d39-b955-45e2-84ad-f4384d17e705\") "
Nov 02 01:19:12.113124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.113097    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-bundle" pod "0fe84d39-b955-45e2-84ad-f4384d17e705" (UID: "0fe84d39-b955-45e2-84ad-f4384d17e705"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:19:12.115091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.115071    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe84d39-b955-45e2-84ad-f4384d17e705-kube-api-access-wsbhx" pod "0fe84d39-b955-45e2-84ad-f4384d17e705" (UID: "0fe84d39-b955-45e2-84ad-f4384d17e705"). InnerVolumeSpecName "kube-api-access-wsbhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:19:12.117114 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.117091    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-util" pod "0fe84d39-b955-45e2-84ad-f4384d17e705" (UID: "0fe84d39-b955-45e2-84ad-f4384d17e705"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:19:12.213865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.213829    4851 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-util\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:19:12.213865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.213859    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsbhx\" (UniqueName: \"kubernetes.io/projected/0fe84d39-b955-45e2-84ad-f4384d17e705-kube-api-access-wsbhx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:19:12.213865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.213869    4851 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fe84d39-b955-45e2-84ad-f4384d17e705-bundle\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:19:12.898788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.898748    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx" event={"ID":"0fe84d39-b955-45e2-84ad-f4384d17e705","Type":"ContainerDied","Data":"174e3afa749363ee3d4ef48164a364212e66d1b81a4f6b2161e3e9970f3e12b9"}
Nov 02 01:19:12.898788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.898765    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644s6jgx"
Nov 02 01:19:12.898989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:12.898800    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="174e3afa749363ee3d4ef48164a364212e66d1b81a4f6b2161e3e9970f3e12b9"
Nov 02 01:19:17.654243 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:19:17.654201    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:19:37.697007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:37.696971    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:38.516073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.516039    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"]
Nov 02 01:19:38.516360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.516346    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0fe84d39-b955-45e2-84ad-f4384d17e705" containerName="util"
Nov 02 01:19:38.516401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.516362    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe84d39-b955-45e2-84ad-f4384d17e705" containerName="util"
Nov 02 01:19:38.516401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.516376    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0fe84d39-b955-45e2-84ad-f4384d17e705" containerName="pull"
Nov 02 01:19:38.516401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.516381    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe84d39-b955-45e2-84ad-f4384d17e705" containerName="pull"
Nov 02 01:19:38.516401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.516388    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0fe84d39-b955-45e2-84ad-f4384d17e705" containerName="extract"
Nov 02 01:19:38.516401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.516393    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe84d39-b955-45e2-84ad-f4384d17e705" containerName="extract"
Nov 02 01:19:38.516553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.516472    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe84d39-b955-45e2-84ad-f4384d17e705" containerName="extract"
Nov 02 01:19:38.519411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.519390    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.549783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.549752    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"]
Nov 02 01:19:38.600145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.600116    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 01:19:38.600145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.600114    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 01:19:38.600145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.600149    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 01:19:38.600448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.600349    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 01:19:38.600448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.600380    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 01:19:38.600448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.600392    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-vj7qc\""
Nov 02 01:19:38.608729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.608705    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/592ace31-a3ce-42ef-b168-64712a32a593-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.608846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.608744    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/592ace31-a3ce-42ef-b168-64712a32a593-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.608846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.608797    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/592ace31-a3ce-42ef-b168-64712a32a593-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.608934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.608846    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8k8\" (UniqueName: \"kubernetes.io/projected/592ace31-a3ce-42ef-b168-64712a32a593-kube-api-access-6v8k8\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.608934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.608908    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/592ace31-a3ce-42ef-b168-64712a32a593-manifests\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.609008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.608949    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/592ace31-a3ce-42ef-b168-64712a32a593-tmp\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.710189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.710151    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/592ace31-a3ce-42ef-b168-64712a32a593-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.710698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.710196    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/592ace31-a3ce-42ef-b168-64712a32a593-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.710698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.710236    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/592ace31-a3ce-42ef-b168-64712a32a593-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.710698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.710260    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8k8\" (UniqueName: \"kubernetes.io/projected/592ace31-a3ce-42ef-b168-64712a32a593-kube-api-access-6v8k8\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.710698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.710321    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/592ace31-a3ce-42ef-b168-64712a32a593-manifests\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.710698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.710371    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/592ace31-a3ce-42ef-b168-64712a32a593-tmp\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.710931 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.710834    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/592ace31-a3ce-42ef-b168-64712a32a593-manifests\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.712952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.712926    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/592ace31-a3ce-42ef-b168-64712a32a593-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.713053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.712976    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/592ace31-a3ce-42ef-b168-64712a32a593-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.713096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.713053    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/592ace31-a3ce-42ef-b168-64712a32a593-tmp\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.713172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.713156    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/592ace31-a3ce-42ef-b168-64712a32a593-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.723140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.723116    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8k8\" (UniqueName: \"kubernetes.io/projected/592ace31-a3ce-42ef-b168-64712a32a593-kube-api-access-6v8k8\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-phmfn\" (UID: \"592ace31-a3ce-42ef-b168-64712a32a593\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:38.834693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:38.834613    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:39.136743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:39.136671    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"]
Nov 02 01:19:39.140402 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:19:39.140375    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592ace31_a3ce_42ef_b168_64712a32a593.slice/crio-3faccb0376d1b8770a4e8b0ed9945b81a2a95502762e075af8a7899be0aec8ba WatchSource:0}: Error finding container 3faccb0376d1b8770a4e8b0ed9945b81a2a95502762e075af8a7899be0aec8ba: Status 404 returned error can't find the container with id 3faccb0376d1b8770a4e8b0ed9945b81a2a95502762e075af8a7899be0aec8ba
Nov 02 01:19:40.002329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:40.002276    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn" event={"ID":"592ace31-a3ce-42ef-b168-64712a32a593","Type":"ContainerStarted","Data":"3faccb0376d1b8770a4e8b0ed9945b81a2a95502762e075af8a7899be0aec8ba"}
Nov 02 01:19:40.166221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:40.166188    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:41.533496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:41.533458    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:41.664897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:41.664857    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:42.116532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:42.116499    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:42.243670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:42.243629    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:45.024853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:45.024820    4851 generic.go:356] "Generic (PLEG): container finished" podID="592ace31-a3ce-42ef-b168-64712a32a593" containerID="aedc158bc441a019aa90a024883538db4c033ca97ae85fe4c96144f969302aa7" exitCode=0
Nov 02 01:19:45.025230 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:45.024859    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn" event={"ID":"592ace31-a3ce-42ef-b168-64712a32a593","Type":"ContainerDied","Data":"aedc158bc441a019aa90a024883538db4c033ca97ae85fe4c96144f969302aa7"}
Nov 02 01:19:45.138078 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:45.138047    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:46.032082 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:46.031993    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn" event={"ID":"592ace31-a3ce-42ef-b168-64712a32a593","Type":"ContainerStarted","Data":"2e0b8542aee7df23b415a8d714746d7f248588db0a7bcce0287437dcccf8016c"}
Nov 02 01:19:46.032570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:46.032137    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:46.077980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:46.077912    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn" podStartSLOduration=1.9336759460000001 podStartE2EDuration="8.077892389s" totalImagesPullingTime="5.715393805s" totalInitContainerRuntime="428.822638ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:19:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:19:46.075703485 +0000 UTC m=+291.380636312" watchObservedRunningTime="2026-11-02 01:19:46.077892389 +0000 UTC m=+291.382825216"
Nov 02 01:19:47.121680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:47.121644    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:47.244817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:47.244785    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:47.655238 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:19:47.655201    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:19:49.109059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:49.109025    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:52.723660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:52.723629    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:54.866972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:54.866945    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:19:54.869217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:54.869199    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:19:55.596233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:55.596197    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:56.700579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:56.700541    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:56.837221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:56.837195    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:56.985469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:56.985387    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:57.038427 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:57.038397    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn"
Nov 02 01:19:57.741565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:57.741513    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:58.497559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:58.497526    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:19:58.678600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:19:58.678571    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:00.774758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:00.774723    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:01.121822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:01.121748    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:01.485728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:01.485697    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:02.918220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:02.918184    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:04.174334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:04.174301    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:05.292098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:05.292016    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:05.944858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:05.944821    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:06.071872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:06.071838    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:06.199111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:06.199025    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:06.527403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:06.527325    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:06.885028 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:06.884952    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:07.293253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:07.293211    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:08.378628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:08.378584    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:08.529685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:08.529645    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:08.723815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:08.723780    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:08.854334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:08.854278    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:10.176999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:10.176962    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:10.310348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:10.310312    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:13.941450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:13.941420    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:14.082851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:14.082821    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:15.950523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:15.950493    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:16.165773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.165742    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:16.286898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.286828    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"]
Nov 02 01:20:16.289982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.289965    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.299462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.299443    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 01:20:16.300252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.300237    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6bnmt\""
Nov 02 01:20:16.300332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.300253    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 01:20:16.306663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.306640    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"]
Nov 02 01:20:16.407252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.407219    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:16.414494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.414466    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bc296a-c669-4abe-8cf0-6eb24154f4a3-tls-certs\") pod \"kube-auth-proxy-6c85779f4b-fx7dq\" (UID: \"13bc296a-c669-4abe-8cf0-6eb24154f4a3\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.414657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.414516    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rmls\" (UniqueName: \"kubernetes.io/projected/13bc296a-c669-4abe-8cf0-6eb24154f4a3-kube-api-access-4rmls\") pod \"kube-auth-proxy-6c85779f4b-fx7dq\" (UID: \"13bc296a-c669-4abe-8cf0-6eb24154f4a3\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.414657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.414581    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13bc296a-c669-4abe-8cf0-6eb24154f4a3-tmp\") pod \"kube-auth-proxy-6c85779f4b-fx7dq\" (UID: \"13bc296a-c669-4abe-8cf0-6eb24154f4a3\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.515585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.515547    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bc296a-c669-4abe-8cf0-6eb24154f4a3-tls-certs\") pod \"kube-auth-proxy-6c85779f4b-fx7dq\" (UID: \"13bc296a-c669-4abe-8cf0-6eb24154f4a3\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.515756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.515610    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rmls\" (UniqueName: \"kubernetes.io/projected/13bc296a-c669-4abe-8cf0-6eb24154f4a3-kube-api-access-4rmls\") pod \"kube-auth-proxy-6c85779f4b-fx7dq\" (UID: \"13bc296a-c669-4abe-8cf0-6eb24154f4a3\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.515756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.515643    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13bc296a-c669-4abe-8cf0-6eb24154f4a3-tmp\") pod \"kube-auth-proxy-6c85779f4b-fx7dq\" (UID: \"13bc296a-c669-4abe-8cf0-6eb24154f4a3\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.517930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.517899    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13bc296a-c669-4abe-8cf0-6eb24154f4a3-tmp\") pod \"kube-auth-proxy-6c85779f4b-fx7dq\" (UID: \"13bc296a-c669-4abe-8cf0-6eb24154f4a3\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.518186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.518168    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bc296a-c669-4abe-8cf0-6eb24154f4a3-tls-certs\") pod \"kube-auth-proxy-6c85779f4b-fx7dq\" (UID: \"13bc296a-c669-4abe-8cf0-6eb24154f4a3\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.529509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.529482    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rmls\" (UniqueName: \"kubernetes.io/projected/13bc296a-c669-4abe-8cf0-6eb24154f4a3-kube-api-access-4rmls\") pod \"kube-auth-proxy-6c85779f4b-fx7dq\" (UID: \"13bc296a-c669-4abe-8cf0-6eb24154f4a3\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.540562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.540494    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:16.601477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.601450    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"
Nov 02 01:20:16.665416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.665278    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:16.880851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.880826    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq"]
Nov 02 01:20:16.883450 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:20:16.883429    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bc296a_c669_4abe_8cf0_6eb24154f4a3.slice/crio-7dda518692280606e311e9739afcea50adec353187ec280ee29745fe2118164f WatchSource:0}: Error finding container 7dda518692280606e311e9739afcea50adec353187ec280ee29745fe2118164f: Status 404 returned error can't find the container with id 7dda518692280606e311e9739afcea50adec353187ec280ee29745fe2118164f
Nov 02 01:20:16.885340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:16.885318    4851 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:20:17.153489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:17.153415    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq" event={"ID":"13bc296a-c669-4abe-8cf0-6eb24154f4a3","Type":"ContainerStarted","Data":"7dda518692280606e311e9739afcea50adec353187ec280ee29745fe2118164f"}
Nov 02 01:20:17.655259 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:20:17.655215    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:20:17.919127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:17.919099    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:18.354575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:18.354494    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:18.704247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:18.704213    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:18.839550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:18.839509    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:18.970671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:18.970598    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:20.166887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:20.166851    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq" event={"ID":"13bc296a-c669-4abe-8cf0-6eb24154f4a3","Type":"ContainerStarted","Data":"c7f05dc7e0061564a4591f31944be36db934dac1343b50be5ac0698620d7964f"}
Nov 02 01:20:20.185722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:20.185678    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c85779f4b-fx7dq" podStartSLOduration=1.118074387 podStartE2EDuration="4.185664763s" totalImagesPullingTime="3.067590376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:20.185091757 +0000 UTC m=+325.490024586" watchObservedRunningTime="2026-11-02 01:20:20.185664763 +0000 UTC m=+325.490597621"
Nov 02 01:20:21.693576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:21.693540    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:21.821789 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:21.821746    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:22.028462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.028382    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:22.175653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.175608    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:22.542233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.542200    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:22.697450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.697413    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:22.797278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.797191    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fss7s"]
Nov 02 01:20:22.800667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.800644    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fss7s"
Nov 02 01:20:22.807139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.806896    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6lfnj\""
Nov 02 01:20:22.808493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.808471    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fss7s"]
Nov 02 01:20:22.867191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.867164    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ckx\" (UniqueName: \"kubernetes.io/projected/da8ae892-8f34-4e93-9669-10f7d2f5cd87-kube-api-access-h7ckx\") pod \"authorino-57f7f8659f-fss7s\" (UID: \"da8ae892-8f34-4e93-9669-10f7d2f5cd87\") " pod="kuadrant-system/authorino-57f7f8659f-fss7s"
Nov 02 01:20:22.874979 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.874953    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 01:20:22.875088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.874982    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 01:20:22.882920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.882891    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:22.968091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.968056    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ckx\" (UniqueName: \"kubernetes.io/projected/da8ae892-8f34-4e93-9669-10f7d2f5cd87-kube-api-access-h7ckx\") pod \"authorino-57f7f8659f-fss7s\" (UID: \"da8ae892-8f34-4e93-9669-10f7d2f5cd87\") " pod="kuadrant-system/authorino-57f7f8659f-fss7s"
Nov 02 01:20:22.986030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:22.986000    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ckx\" (UniqueName: \"kubernetes.io/projected/da8ae892-8f34-4e93-9669-10f7d2f5cd87-kube-api-access-h7ckx\") pod \"authorino-57f7f8659f-fss7s\" (UID: \"da8ae892-8f34-4e93-9669-10f7d2f5cd87\") " pod="kuadrant-system/authorino-57f7f8659f-fss7s"
Nov 02 01:20:23.019170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:23.019138    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:23.120501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:23.120431    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fss7s"
Nov 02 01:20:23.391503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:23.391430    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fss7s"]
Nov 02 01:20:23.394175 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:20:23.394148    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda8ae892_8f34_4e93_9669_10f7d2f5cd87.slice/crio-0ee41bad23e0b7fba6f1c301979a6f3e1b148499ae1c766cf3d5409e97eddc90 WatchSource:0}: Error finding container 0ee41bad23e0b7fba6f1c301979a6f3e1b148499ae1c766cf3d5409e97eddc90: Status 404 returned error can't find the container with id 0ee41bad23e0b7fba6f1c301979a6f3e1b148499ae1c766cf3d5409e97eddc90
Nov 02 01:20:24.185040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:24.185010    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fss7s" event={"ID":"da8ae892-8f34-4e93-9669-10f7d2f5cd87","Type":"ContainerStarted","Data":"0ee41bad23e0b7fba6f1c301979a6f3e1b148499ae1c766cf3d5409e97eddc90"}
Nov 02 01:20:28.091805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:28.091761    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:28.213695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:28.213660    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:29.032192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.032149    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:29.170601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.170571    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:29.215828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.215787    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fss7s" event={"ID":"da8ae892-8f34-4e93-9669-10f7d2f5cd87","Type":"ContainerStarted","Data":"7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498"}
Nov 02 01:20:29.240743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.240681    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-fss7s" podStartSLOduration=2.161625516 podStartE2EDuration="7.240663283s" totalImagesPullingTime="5.079037767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:29.240195682 +0000 UTC m=+334.545128509" watchObservedRunningTime="2026-11-02 01:20:29.240663283 +0000 UTC m=+334.545596109"
Nov 02 01:20:29.389006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.388924    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"]
Nov 02 01:20:29.392380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.392360    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.402133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.402110    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 01:20:29.402337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.402321    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 01:20:29.402423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.402391    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 01:20:29.402423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.402391    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 01:20:29.402535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.402430    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-cjlrx\""
Nov 02 01:20:29.402535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.402476    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:20:29.403077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.403056    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"]
Nov 02 01:20:29.418083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.418054    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-metrics-certs\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.418239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.418100    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-cert\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.418239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.418132    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgvtd\" (UniqueName: \"kubernetes.io/projected/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-kube-api-access-jgvtd\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.418344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.418320    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-manager-config\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.519399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.519361    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-metrics-certs\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.519616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.519410    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-cert\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.519616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.519438    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgvtd\" (UniqueName: \"kubernetes.io/projected/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-kube-api-access-jgvtd\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.519616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.519511    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-manager-config\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.520326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.520188    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-manager-config\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.522041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.522022    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-cert\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.522216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.522197    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-metrics-certs\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.531342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.531318    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgvtd\" (UniqueName: \"kubernetes.io/projected/8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b-kube-api-access-jgvtd\") pod \"jobset-controller-manager-5b9f7dd4f5-84bht\" (UID: \"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b\") " pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.703773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.703743    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:29.980013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:29.979991    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"]
Nov 02 01:20:29.982112 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:20:29.982084    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8286a9d3_cdfb_4e96_b7cf_e61f6f5c2f8b.slice/crio-6fc73e2174cf585d2484124ffc46d225901db95d07cc55120b1306384ae71b7d WatchSource:0}: Error finding container 6fc73e2174cf585d2484124ffc46d225901db95d07cc55120b1306384ae71b7d: Status 404 returned error can't find the container with id 6fc73e2174cf585d2484124ffc46d225901db95d07cc55120b1306384ae71b7d
Nov 02 01:20:30.221098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:30.221064    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht" event={"ID":"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b","Type":"ContainerStarted","Data":"6fc73e2174cf585d2484124ffc46d225901db95d07cc55120b1306384ae71b7d"}
Nov 02 01:20:30.916688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:30.916652    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fss7s"]
Nov 02 01:20:31.224959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:31.224849    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-fss7s" podUID="da8ae892-8f34-4e93-9669-10f7d2f5cd87" containerName="authorino" containerID="cri-o://7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498" gracePeriod=30
Nov 02 01:20:31.461121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:31.461099    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fss7s"
Nov 02 01:20:31.536323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:31.536236    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8ae892-8f34-4e93-9669-10f7d2f5cd87-kube-api-access-h7ckx\" (UniqueName: \"kubernetes.io/projected/da8ae892-8f34-4e93-9669-10f7d2f5cd87-kube-api-access-h7ckx\") pod \"da8ae892-8f34-4e93-9669-10f7d2f5cd87\" (UID: \"da8ae892-8f34-4e93-9669-10f7d2f5cd87\") "
Nov 02 01:20:31.538668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:31.538639    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8ae892-8f34-4e93-9669-10f7d2f5cd87-kube-api-access-h7ckx" pod "da8ae892-8f34-4e93-9669-10f7d2f5cd87" (UID: "da8ae892-8f34-4e93-9669-10f7d2f5cd87"). InnerVolumeSpecName "kube-api-access-h7ckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:20:31.637661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:31.637631    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7ckx\" (UniqueName: \"kubernetes.io/projected/da8ae892-8f34-4e93-9669-10f7d2f5cd87-kube-api-access-h7ckx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:20:32.229982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:32.229946    4851 generic.go:356] "Generic (PLEG): container finished" podID="da8ae892-8f34-4e93-9669-10f7d2f5cd87" containerID="7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498" exitCode=0
Nov 02 01:20:32.230465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:32.229991    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-fss7s"
Nov 02 01:20:32.230465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:32.230013    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fss7s" event={"ID":"da8ae892-8f34-4e93-9669-10f7d2f5cd87","Type":"ContainerDied","Data":"7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498"}
Nov 02 01:20:32.230465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:32.230053    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-fss7s" event={"ID":"da8ae892-8f34-4e93-9669-10f7d2f5cd87","Type":"ContainerDied","Data":"0ee41bad23e0b7fba6f1c301979a6f3e1b148499ae1c766cf3d5409e97eddc90"}
Nov 02 01:20:32.230465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:32.230077    4851 scope.go:122] "RemoveContainer" containerID="7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498"
Nov 02 01:20:32.239004 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:32.238987    4851 scope.go:122] "RemoveContainer" containerID="7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498"
Nov 02 01:20:32.239276 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:20:32.239258    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498\": container with ID starting with 7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498 not found: ID does not exist" containerID="7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498"
Nov 02 01:20:32.239364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:32.239312    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498"} err="failed to get container status \"7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498\": rpc error: code = NotFound desc = could not find container \"7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498\": container with ID starting with 7d1156627c7721b9f37e9884eba796d3c732a997c8a48625aabd688e06123498 not found: ID does not exist"
Nov 02 01:20:32.252100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:32.252073    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fss7s"]
Nov 02 01:20:32.254307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:32.254273    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-fss7s"]
Nov 02 01:20:32.844922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:32.844882    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8ae892-8f34-4e93-9669-10f7d2f5cd87" path="/var/lib/kubelet/pods/da8ae892-8f34-4e93-9669-10f7d2f5cd87/volumes"
Nov 02 01:20:34.240897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:34.240861    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht" event={"ID":"8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b","Type":"ContainerStarted","Data":"b420d6e44bc4d6809f74cf07fdc4149ad9b1ff221361c73bd4e81dbece557cff"}
Nov 02 01:20:34.241281 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:34.240999    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:34.262378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:34.262278    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht" podStartSLOduration=1.6919147479999999 podStartE2EDuration="5.26226679s" totalImagesPullingTime="3.570352042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:34.260989516 +0000 UTC m=+339.565922354" watchObservedRunningTime="2026-11-02 01:20:34.26226679 +0000 UTC m=+339.567199616"
Nov 02 01:20:34.753644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:34.753601    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:34.886716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:34.886683    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:38.339266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:38.339228    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:38.475667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:38.475626    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:38.603991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:38.603899    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:38.763749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:38.763712    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:38.943963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:38.943935    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:39.403877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:39.403794    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:42.726480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:42.726446    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:42.847873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:42.847836    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:45.190614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:45.190578    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:45.249652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:45.249620    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht"
Nov 02 01:20:47.654403 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:20:47.654363    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:20:48.594954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:48.594921    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:48.725247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:48.725214    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:58.933349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:58.933312    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:59.063412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:59.063376    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:20:59.188479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:20:59.188397    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:02.660168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:02.660128    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:02.781324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:02.781276    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:03.851521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:03.851489    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:03.979483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:03.979452    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:08.536056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:08.536023    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:08.688039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:08.688004    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:14.576842 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:14.576767    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:14.728002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:14.727971    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:14.856218 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:14.856145    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:14.919200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:14.919174    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"]
Nov 02 01:21:14.919544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:14.919527    4851 cpu_manager.go:414] "containerMap: removing container" podUID="da8ae892-8f34-4e93-9669-10f7d2f5cd87" containerName="authorino"
Nov 02 01:21:14.919625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:14.919547    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8ae892-8f34-4e93-9669-10f7d2f5cd87" containerName="authorino"
Nov 02 01:21:14.919625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:14.919603    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8ae892-8f34-4e93-9669-10f7d2f5cd87" containerName="authorino"
Nov 02 01:21:14.924777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:14.924760    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:14.935746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:14.935724    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"]
Nov 02 01:21:14.998323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:14.998265    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:15.108181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.108098    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a77a15-50e0-43d2-b455-459f74de3250-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.108181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.108136    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.108181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.108155    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4t8\" (UniqueName: \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-kube-api-access-sh4t8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.108425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.108274    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.108425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.108320    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.108425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.108378    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.108425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.108412    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.208827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.208788    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.208994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.208846    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a77a15-50e0-43d2-b455-459f74de3250-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.208994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.208867    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.208994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.208886    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4t8\" (UniqueName: \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-kube-api-access-sh4t8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.208994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.208905    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.208994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.208920    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.208994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.208948    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.209753 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.209717    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.211495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.211473    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.211668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.211652    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a77a15-50e0-43d2-b455-459f74de3250-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.211827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.211807    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.211923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.211907    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.235404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.235371    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.235542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.235513    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4t8\" (UniqueName: \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-kube-api-access-sh4t8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nnr4k\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.238202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.238186    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:15.532486 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.532448    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"]
Nov 02 01:21:15.540412 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:21:15.540384    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a77a15_50e0_43d2_b455_459f74de3250.slice/crio-e56f9e456d291ba7cd2323b99b8b23bfdd6030330d1d7c41483e7e0744356dfb WatchSource:0}: Error finding container e56f9e456d291ba7cd2323b99b8b23bfdd6030330d1d7c41483e7e0744356dfb: Status 404 returned error can't find the container with id e56f9e456d291ba7cd2323b99b8b23bfdd6030330d1d7c41483e7e0744356dfb
Nov 02 01:21:15.542419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.542385    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:21:15.542505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.542451    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:21:15.876487 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:15.876401    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:16.012389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.012355    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:16.142460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.142385    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:16.414485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.414445    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k" event={"ID":"c1a77a15-50e0-43d2-b455-459f74de3250","Type":"ContainerStarted","Data":"6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555"}
Nov 02 01:21:16.414674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.414533    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k" event={"ID":"c1a77a15-50e0-43d2-b455-459f74de3250","Type":"ContainerStarted","Data":"e56f9e456d291ba7cd2323b99b8b23bfdd6030330d1d7c41483e7e0744356dfb"}
Nov 02 01:21:16.414741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.414669    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:16.416460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.416440    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:21:16.439352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.439278    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k" podStartSLOduration=2.439265186 podStartE2EDuration="2.439265186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:16.437533665 +0000 UTC m=+381.742466492" watchObservedRunningTime="2026-11-02 01:21:16.439265186 +0000 UTC m=+381.744198011"
Nov 02 01:21:16.492796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.492761    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"]
Nov 02 01:21:16.493042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.493010    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629" podUID="3b1fa953-1b1b-4590-921f-68d7fa99e9a9" containerName="discovery" containerID="cri-o://4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7" gracePeriod=30
Nov 02 01:21:16.738110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.738086    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:21:16.921572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.921539    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-ca-configmap\") pod \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") "
Nov 02 01:21:16.922088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.921589    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-kubeconfig\") pod \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") "
Nov 02 01:21:16.922088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.921630    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-local-certs\") pod \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") "
Nov 02 01:21:16.922088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.921689    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-cacerts\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-cacerts\") pod \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") "
Nov 02 01:21:16.922088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.921723    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-dns-cert\") pod \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") "
Nov 02 01:21:16.922088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.921772    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-token\" (UniqueName: \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-token\") pod \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") "
Nov 02 01:21:16.922088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.921797    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-kube-api-access-zt4mw\" (UniqueName: \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-kube-api-access-zt4mw\") pod \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\" (UID: \"3b1fa953-1b1b-4590-921f-68d7fa99e9a9\") "
Nov 02 01:21:16.922088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.922002    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-ca-configmap" pod "3b1fa953-1b1b-4590-921f-68d7fa99e9a9" (UID: "3b1fa953-1b1b-4590-921f-68d7fa99e9a9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:21:16.924629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.924595    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-kubeconfig" pod "3b1fa953-1b1b-4590-921f-68d7fa99e9a9" (UID: "3b1fa953-1b1b-4590-921f-68d7fa99e9a9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:21:16.924629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.924611    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-dns-cert" pod "3b1fa953-1b1b-4590-921f-68d7fa99e9a9" (UID: "3b1fa953-1b1b-4590-921f-68d7fa99e9a9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:21:16.924772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.924628    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-cacerts" pod "3b1fa953-1b1b-4590-921f-68d7fa99e9a9" (UID: "3b1fa953-1b1b-4590-921f-68d7fa99e9a9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:21:16.924772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.924602    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-kube-api-access-zt4mw" pod "3b1fa953-1b1b-4590-921f-68d7fa99e9a9" (UID: "3b1fa953-1b1b-4590-921f-68d7fa99e9a9"). InnerVolumeSpecName "kube-api-access-zt4mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:21:16.924772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.924679    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-token" pod "3b1fa953-1b1b-4590-921f-68d7fa99e9a9" (UID: "3b1fa953-1b1b-4590-921f-68d7fa99e9a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:21:16.924869 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:16.924846    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-local-certs" pod "3b1fa953-1b1b-4590-921f-68d7fa99e9a9" (UID: "3b1fa953-1b1b-4590-921f-68d7fa99e9a9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:21:17.023095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.022992    4851 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-kubeconfig\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:21:17.023095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.023030    4851 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-local-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:21:17.023095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.023042    4851 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-cacerts\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:21:17.023095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.023056    4851 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-dns-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:21:17.023095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.023069    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:21:17.023095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.023082    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt4mw\" (UniqueName: \"kubernetes.io/projected/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-kube-api-access-zt4mw\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:21:17.023095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.023097    4851 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3b1fa953-1b1b-4590-921f-68d7fa99e9a9-istio-csr-ca-configmap\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:21:17.420585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.420546    4851 generic.go:356] "Generic (PLEG): container finished" podID="3b1fa953-1b1b-4590-921f-68d7fa99e9a9" containerID="4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7" exitCode=0
Nov 02 01:21:17.420760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.420592    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"
Nov 02 01:21:17.420760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.420629    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629" event={"ID":"3b1fa953-1b1b-4590-921f-68d7fa99e9a9","Type":"ContainerDied","Data":"4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7"}
Nov 02 01:21:17.420760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.420675    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629" event={"ID":"3b1fa953-1b1b-4590-921f-68d7fa99e9a9","Type":"ContainerDied","Data":"eb896e463cb7bcf663e669805405d20f0b6e6178c54917bd806509bb5e198001"}
Nov 02 01:21:17.420760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.420695    4851 scope.go:122] "RemoveContainer" containerID="4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7"
Nov 02 01:21:17.429353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.429338    4851 scope.go:122] "RemoveContainer" containerID="4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7"
Nov 02 01:21:17.429614 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:21:17.429594    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7\": container with ID starting with 4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7 not found: ID does not exist" containerID="4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7"
Nov 02 01:21:17.429679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.429626    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7"} err="failed to get container status \"4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7\": rpc error: code = NotFound desc = could not find container \"4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7\": container with ID starting with 4acd1a50a64d3453118a5b14e2ac39aae22b67bb99a173b4fde12760885926c7 not found: ID does not exist"
Nov 02 01:21:17.442637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.442616    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"]
Nov 02 01:21:17.447093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:17.447073    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2c629"]
Nov 02 01:21:17.654125 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:21:17.654086    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:21:18.844428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:18.844399    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1fa953-1b1b-4590-921f-68d7fa99e9a9" path="/var/lib/kubelet/pods/3b1fa953-1b1b-4590-921f-68d7fa99e9a9/volumes"
Nov 02 01:21:30.103714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:30.103677    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:30.226057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:30.226013    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:31.421419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:31.421385    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:31.552984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:31.552954    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:31.692518 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:31.692431    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:41.681700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.681621    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:41.914873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.914820    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:41.951248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.951169    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-6xjhx"]
Nov 02 01:21:41.951496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.951484    4851 cpu_manager.go:414] "containerMap: removing container" podUID="3b1fa953-1b1b-4590-921f-68d7fa99e9a9" containerName="discovery"
Nov 02 01:21:41.951550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.951497    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1fa953-1b1b-4590-921f-68d7fa99e9a9" containerName="discovery"
Nov 02 01:21:41.951592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.951583    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1fa953-1b1b-4590-921f-68d7fa99e9a9" containerName="discovery"
Nov 02 01:21:41.954388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.954373    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-6xjhx"
Nov 02 01:21:41.961483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.961444    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6lfnj\""
Nov 02 01:21:41.962599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.962570    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-6xjhx"]
Nov 02 01:21:41.964700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.964678    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 01:21:41.964700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.964687    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 01:21:41.964836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.964697    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 01:21:41.998822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:41.998777    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-6xjhx"]
Nov 02 01:21:41.999032 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:21:41.999008    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sq9z2 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-6xjhx" podUID="c60a91b5-abd9-44a2-89d6-3e9cc8fa3629"
Nov 02 01:21:42.007700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.007678    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq9z2\" (UniqueName: \"kubernetes.io/projected/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-kube-api-access-sq9z2\") pod \"authorino-6dd76875b5-6xjhx\" (UID: \"c60a91b5-abd9-44a2-89d6-3e9cc8fa3629\") " pod="kuadrant-system/authorino-6dd76875b5-6xjhx"
Nov 02 01:21:42.007776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.007722    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-tls-cert\") pod \"authorino-6dd76875b5-6xjhx\" (UID: \"c60a91b5-abd9-44a2-89d6-3e9cc8fa3629\") " pod="kuadrant-system/authorino-6dd76875b5-6xjhx"
Nov 02 01:21:42.108806 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.108764    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-tls-cert\") pod \"authorino-6dd76875b5-6xjhx\" (UID: \"c60a91b5-abd9-44a2-89d6-3e9cc8fa3629\") " pod="kuadrant-system/authorino-6dd76875b5-6xjhx"
Nov 02 01:21:42.108988 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.108847    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq9z2\" (UniqueName: \"kubernetes.io/projected/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-kube-api-access-sq9z2\") pod \"authorino-6dd76875b5-6xjhx\" (UID: \"c60a91b5-abd9-44a2-89d6-3e9cc8fa3629\") " pod="kuadrant-system/authorino-6dd76875b5-6xjhx"
Nov 02 01:21:42.111464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.111444    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-tls-cert\") pod \"authorino-6dd76875b5-6xjhx\" (UID: \"c60a91b5-abd9-44a2-89d6-3e9cc8fa3629\") " pod="kuadrant-system/authorino-6dd76875b5-6xjhx"
Nov 02 01:21:42.121039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.121009    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq9z2\" (UniqueName: \"kubernetes.io/projected/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-kube-api-access-sq9z2\") pod \"authorino-6dd76875b5-6xjhx\" (UID: \"c60a91b5-abd9-44a2-89d6-3e9cc8fa3629\") " pod="kuadrant-system/authorino-6dd76875b5-6xjhx"
Nov 02 01:21:42.379029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.378948    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b5bd4d575-bjcs6"]
Nov 02 01:21:42.382491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.382468    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b5bd4d575-bjcs6"
Nov 02 01:21:42.394705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.394671    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b5bd4d575-bjcs6"]
Nov 02 01:21:42.411422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.411377    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmbgm\" (UniqueName: \"kubernetes.io/projected/3f57c479-0991-432b-ac66-c4cb9c71d16f-kube-api-access-wmbgm\") pod \"authorino-5b5bd4d575-bjcs6\" (UID: \"3f57c479-0991-432b-ac66-c4cb9c71d16f\") " pod="kuadrant-system/authorino-5b5bd4d575-bjcs6"
Nov 02 01:21:42.411599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.411505    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f57c479-0991-432b-ac66-c4cb9c71d16f-tls-cert\") pod \"authorino-5b5bd4d575-bjcs6\" (UID: \"3f57c479-0991-432b-ac66-c4cb9c71d16f\") " pod="kuadrant-system/authorino-5b5bd4d575-bjcs6"
Nov 02 01:21:42.512577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.512532    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f57c479-0991-432b-ac66-c4cb9c71d16f-tls-cert\") pod \"authorino-5b5bd4d575-bjcs6\" (UID: \"3f57c479-0991-432b-ac66-c4cb9c71d16f\") " pod="kuadrant-system/authorino-5b5bd4d575-bjcs6"
Nov 02 01:21:42.512732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.512598    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmbgm\" (UniqueName: \"kubernetes.io/projected/3f57c479-0991-432b-ac66-c4cb9c71d16f-kube-api-access-wmbgm\") pod \"authorino-5b5bd4d575-bjcs6\" (UID: \"3f57c479-0991-432b-ac66-c4cb9c71d16f\") " pod="kuadrant-system/authorino-5b5bd4d575-bjcs6"
Nov 02 01:21:42.515237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.515208    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f57c479-0991-432b-ac66-c4cb9c71d16f-tls-cert\") pod \"authorino-5b5bd4d575-bjcs6\" (UID: \"3f57c479-0991-432b-ac66-c4cb9c71d16f\") " pod="kuadrant-system/authorino-5b5bd4d575-bjcs6"
Nov 02 01:21:42.522905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.522878    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-6xjhx"
Nov 02 01:21:42.524969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.524945    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmbgm\" (UniqueName: \"kubernetes.io/projected/3f57c479-0991-432b-ac66-c4cb9c71d16f-kube-api-access-wmbgm\") pod \"authorino-5b5bd4d575-bjcs6\" (UID: \"3f57c479-0991-432b-ac66-c4cb9c71d16f\") " pod="kuadrant-system/authorino-5b5bd4d575-bjcs6"
Nov 02 01:21:42.553134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.553099    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-6xjhx"
Nov 02 01:21:42.613191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.613157    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-tls-cert\" (UniqueName: \"kubernetes.io/secret/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-tls-cert\") pod \"c60a91b5-abd9-44a2-89d6-3e9cc8fa3629\" (UID: \"c60a91b5-abd9-44a2-89d6-3e9cc8fa3629\") "
Nov 02 01:21:42.613414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.613212    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-kube-api-access-sq9z2\" (UniqueName: \"kubernetes.io/projected/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-kube-api-access-sq9z2\") pod \"c60a91b5-abd9-44a2-89d6-3e9cc8fa3629\" (UID: \"c60a91b5-abd9-44a2-89d6-3e9cc8fa3629\") "
Nov 02 01:21:42.615560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.615531    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-tls-cert" pod "c60a91b5-abd9-44a2-89d6-3e9cc8fa3629" (UID: "c60a91b5-abd9-44a2-89d6-3e9cc8fa3629"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:21:42.615675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.615573    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-kube-api-access-sq9z2" pod "c60a91b5-abd9-44a2-89d6-3e9cc8fa3629" (UID: "c60a91b5-abd9-44a2-89d6-3e9cc8fa3629"). InnerVolumeSpecName "kube-api-access-sq9z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:21:42.697636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.697598    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b5bd4d575-bjcs6"
Nov 02 01:21:42.714100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.714065    4851 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-tls-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:21:42.714213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.714105    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq9z2\" (UniqueName: \"kubernetes.io/projected/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629-kube-api-access-sq9z2\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:21:42.970535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:42.970468    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b5bd4d575-bjcs6"]
Nov 02 01:21:42.973273 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:21:42.973247    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f57c479_0991_432b_ac66_c4cb9c71d16f.slice/crio-5564bd39465e325d7dfbd75d1c97f418631648ce286336ebee6b8224da3e02de WatchSource:0}: Error finding container 5564bd39465e325d7dfbd75d1c97f418631648ce286336ebee6b8224da3e02de: Status 404 returned error can't find the container with id 5564bd39465e325d7dfbd75d1c97f418631648ce286336ebee6b8224da3e02de
Nov 02 01:21:43.529321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:43.529263    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b5bd4d575-bjcs6" event={"ID":"3f57c479-0991-432b-ac66-c4cb9c71d16f","Type":"ContainerStarted","Data":"5564bd39465e325d7dfbd75d1c97f418631648ce286336ebee6b8224da3e02de"}
Nov 02 01:21:43.529501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:43.529272    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-6xjhx"
Nov 02 01:21:43.560735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:43.560703    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-6xjhx"]
Nov 02 01:21:43.567121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:43.567096    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-6xjhx"]
Nov 02 01:21:44.844055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:44.844022    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60a91b5-abd9-44a2-89d6-3e9cc8fa3629" path="/var/lib/kubelet/pods/c60a91b5-abd9-44a2-89d6-3e9cc8fa3629/volumes"
Nov 02 01:21:45.246958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:45.246926    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:21:47.654232 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:21:47.654192    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:21:51.563566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:51.563526    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b5bd4d575-bjcs6" event={"ID":"3f57c479-0991-432b-ac66-c4cb9c71d16f","Type":"ContainerStarted","Data":"5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3"}
Nov 02 01:21:51.581898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:21:51.581847    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b5bd4d575-bjcs6" podStartSLOduration=1.707229058 podStartE2EDuration="9.581830712s" totalImagesPullingTime="7.874601654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:51.580189101 +0000 UTC m=+416.885121928" watchObservedRunningTime="2026-11-02 01:21:51.581830712 +0000 UTC m=+416.886763541"
Nov 02 01:22:17.654867 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:22:17.654827    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:22:45.300327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:22:45.300276    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:22:46.272496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:22:46.272459    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:22:46.411330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:22:46.411270    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:22:47.654268 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:22:47.654233    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:23:04.525892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:04.525852    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:06.285414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.285326    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"]
Nov 02 01:23:06.289157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.289136    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.298368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.298215    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 01:23:06.299215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.299196    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 01:23:06.299359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.299334    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5855b\""
Nov 02 01:23:06.299359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.299233    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 01:23:06.300345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.300325    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"]
Nov 02 01:23:06.371409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.371374    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:06.404117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.404079    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jssz\" (UniqueName: \"kubernetes.io/projected/d46b2d7a-fabe-4466-818c-9fc511dbd10e-kube-api-access-7jssz\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.404309 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.404133    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.404309 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.404207    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.404408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.404305    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d46b2d7a-fabe-4466-818c-9fc511dbd10e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.404408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.404346    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.404408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.404366    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.505829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.505781    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.505829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.505829    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.506084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.505957    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jssz\" (UniqueName: \"kubernetes.io/projected/d46b2d7a-fabe-4466-818c-9fc511dbd10e-kube-api-access-7jssz\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.506084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.506003    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.506084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.506030    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.506084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.506072    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d46b2d7a-fabe-4466-818c-9fc511dbd10e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.506341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.506146    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.506341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.506228    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.506417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.506380    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.508495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.508467    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d46b2d7a-fabe-4466-818c-9fc511dbd10e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.508698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.508682    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d46b2d7a-fabe-4466-818c-9fc511dbd10e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.517916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.517896    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jssz\" (UniqueName: \"kubernetes.io/projected/d46b2d7a-fabe-4466-818c-9fc511dbd10e-kube-api-access-7jssz\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-srgl9\" (UID: \"d46b2d7a-fabe-4466-818c-9fc511dbd10e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.603431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.603344    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:06.898922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:06.898895    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"]
Nov 02 01:23:06.900235 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:23:06.900203    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46b2d7a_fabe_4466_818c_9fc511dbd10e.slice/crio-d211a078dc54c263550da124e8e695987f500a5db955ad0a91a74ac014eb9e30 WatchSource:0}: Error finding container d211a078dc54c263550da124e8e695987f500a5db955ad0a91a74ac014eb9e30: Status 404 returned error can't find the container with id d211a078dc54c263550da124e8e695987f500a5db955ad0a91a74ac014eb9e30
Nov 02 01:23:07.079864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:07.079830    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:07.879148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:07.879105    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9" event={"ID":"d46b2d7a-fabe-4466-818c-9fc511dbd10e","Type":"ContainerStarted","Data":"d211a078dc54c263550da124e8e695987f500a5db955ad0a91a74ac014eb9e30"}
Nov 02 01:23:08.884662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:08.884630    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9" event={"ID":"d46b2d7a-fabe-4466-818c-9fc511dbd10e","Type":"ContainerStarted","Data":"6ac3fef69390ea2106c14997476f618b61e49759706386542f7850306f514bd6"}
Nov 02 01:23:08.905368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:08.905307    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9" podStartSLOduration=1.6786222830000002 podStartE2EDuration="2.905276505s" totalImagesPullingTime="1.226654222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:08.904041845 +0000 UTC m=+494.208974673" watchObservedRunningTime="2026-11-02 01:23:08.905276505 +0000 UTC m=+494.210209325"
Nov 02 01:23:09.125375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.125337    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"]
Nov 02 01:23:09.128641 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.128620    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.132149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.132126    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 02 01:23:09.140530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.140509    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"]
Nov 02 01:23:09.208457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.208418    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:09.245378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.245192    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.245378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.245252    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.245378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.245374    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbpmj\" (UniqueName: \"kubernetes.io/projected/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-kube-api-access-sbpmj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.245751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.245438    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.245751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.245521    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.245751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.245562    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.346432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.346398    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.346432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.346432    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.346672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.346476    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbpmj\" (UniqueName: \"kubernetes.io/projected/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-kube-api-access-sbpmj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.346672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.346501    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.346672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.346536    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.346672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.346574    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.347000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.346977    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.347080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.347014    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.347080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.347032    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.349039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.349013    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.349343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.349323    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.358787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.358761    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbpmj\" (UniqueName: \"kubernetes.io/projected/d52b9b8d-07d7-49d9-b7e0-92cd937b26c6-kube-api-access-sbpmj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr\" (UID: \"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.444264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.444234    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:09.604105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.604070    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:09.719535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.719503    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"]
Nov 02 01:23:09.722180 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:23:09.722154    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52b9b8d_07d7_49d9_b7e0_92cd937b26c6.slice/crio-e642039932b1c8616007424399bb33e325a4150eeb5a4c0357035269987e0917 WatchSource:0}: Error finding container e642039932b1c8616007424399bb33e325a4150eeb5a4c0357035269987e0917: Status 404 returned error can't find the container with id e642039932b1c8616007424399bb33e325a4150eeb5a4c0357035269987e0917
Nov 02 01:23:09.892733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.892695    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr" event={"ID":"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6","Type":"ContainerStarted","Data":"e642039932b1c8616007424399bb33e325a4150eeb5a4c0357035269987e0917"}
Nov 02 01:23:09.990192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.990112    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jgzjl"]
Nov 02 01:23:09.994731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:09.994692    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.006661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.006633    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jgzjl"]
Nov 02 01:23:10.006861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.006845    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 01:23:10.076137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.076112    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:10.152993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.152956    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.153185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.153021    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.153185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.153115    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437397c1-b8b9-4a7a-90db-17411343e18e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.153284 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.153194    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-home\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.153284 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.153265    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq4lv\" (UniqueName: \"kubernetes.io/projected/437397c1-b8b9-4a7a-90db-17411343e18e-kube-api-access-nq4lv\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.153393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.153335    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.253897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.253804    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.253897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.253854    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.253897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.253878    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437397c1-b8b9-4a7a-90db-17411343e18e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.254172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.253919    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-home\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.254172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.253943    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4lv\" (UniqueName: \"kubernetes.io/projected/437397c1-b8b9-4a7a-90db-17411343e18e-kube-api-access-nq4lv\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.254282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.254242    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.254480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.254434    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-home\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.254577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.254552    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.254640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.254567    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.256669 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.256650    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437397c1-b8b9-4a7a-90db-17411343e18e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.256763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.256684    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437397c1-b8b9-4a7a-90db-17411343e18e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.266579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.266555    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4lv\" (UniqueName: \"kubernetes.io/projected/437397c1-b8b9-4a7a-90db-17411343e18e-kube-api-access-nq4lv\") pod \"echo-simulator-kserve-64b6f8f96c-jgzjl\" (UID: \"437397c1-b8b9-4a7a-90db-17411343e18e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.314868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.314825    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:10.597689 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.597619    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jgzjl"]
Nov 02 01:23:10.601249 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:23:10.601216    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437397c1_b8b9_4a7a_90db_17411343e18e.slice/crio-e43b04b9cccfbb21848fade28a49d7cd915d702afc2da2a2870d6cc9dae009ce WatchSource:0}: Error finding container e43b04b9cccfbb21848fade28a49d7cd915d702afc2da2a2870d6cc9dae009ce: Status 404 returned error can't find the container with id e43b04b9cccfbb21848fade28a49d7cd915d702afc2da2a2870d6cc9dae009ce
Nov 02 01:23:10.683029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.682990    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:10.899552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.899459    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr" event={"ID":"d52b9b8d-07d7-49d9-b7e0-92cd937b26c6","Type":"ContainerStarted","Data":"5761d0081e186b5104a045831107cb208e8455ec2962997535aeff1547978264"}
Nov 02 01:23:10.905341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.903775    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl" event={"ID":"437397c1-b8b9-4a7a-90db-17411343e18e","Type":"ContainerStarted","Data":"e43b04b9cccfbb21848fade28a49d7cd915d702afc2da2a2870d6cc9dae009ce"}
Nov 02 01:23:10.924108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:10.924051    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr" podStartSLOduration=1.636681162 podStartE2EDuration="1.924031769s" totalImagesPullingTime="287.350607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:10.921877563 +0000 UTC m=+496.226810383" watchObservedRunningTime="2026-11-02 01:23:10.924031769 +0000 UTC m=+496.228964596"
Nov 02 01:23:11.284059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:11.284024    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:11.444694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:11.444618    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:12.492356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:12.492320    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:12.914780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:12.914741    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl" event={"ID":"437397c1-b8b9-4a7a-90db-17411343e18e","Type":"ContainerStarted","Data":"69f98a3e6cdf406eeeab9ae62e7185f382d915b28617ee418f00f637196e73b3"}
Nov 02 01:23:12.937219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:12.937148    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl" podStartSLOduration=2.158661111 podStartE2EDuration="3.937126877s" totalImagesPullingTime="1.778465766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:12.934637643 +0000 UTC m=+498.239570474" watchObservedRunningTime="2026-11-02 01:23:12.937126877 +0000 UTC m=+498.242059704"
Nov 02 01:23:13.315496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:13.315397    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:16.604470 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:16.604438    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:16.616756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:16.616731    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:16.943263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:16.943235    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9"
Nov 02 01:23:17.654699 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:23:17.654658    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:23:19.445128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:19.445100    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:19.457639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:19.457616    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:19.830237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:19.830163    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:19.955381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:19.955358    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:23:19.958237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:19.958213    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:20.315022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:20.314985    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:20.327856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:20.327833    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:20.959170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:20.959146    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:23:22.352893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:22.352857    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:22.473620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:22.473588    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:23.167223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.167191    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"]
Nov 02 01:23:23.170878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.170860    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.175252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.175229    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-48jtq\""
Nov 02 01:23:23.185126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.185103    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"]
Nov 02 01:23:23.274005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.273974    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.274189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.274014    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.274189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.274038    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.274189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.274110    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.274393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.274266    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.274393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.274334    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.274526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.274432    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.274526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.274453    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lltc7\" (UniqueName: \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-kube-api-access-lltc7\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.274526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.274474    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.375536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.375501    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.375536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.375534    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.375557    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.375573    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.375634    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.375650    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lltc7\" (UniqueName: \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-kube-api-access-lltc7\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.375671    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.375727    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.375776    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.375996    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.376072    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.376164    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.376212    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.376523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.376490    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.378052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.378032    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.378485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.378464    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.388840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.388813    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lltc7\" (UniqueName: \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-kube-api-access-lltc7\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.388840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.388834    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7f78h\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.485170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.485083    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:23.764829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.764742    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"]
Nov 02 01:23:23.767078 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:23:23.767048    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8e15b0_c38a_471b_b7e5_3cb8987f49dd.slice/crio-b06b37e325451ed9baedb6d566468d8b94f922d8a3e05d3476144cd7e9523bb0 WatchSource:0}: Error finding container b06b37e325451ed9baedb6d566468d8b94f922d8a3e05d3476144cd7e9523bb0: Status 404 returned error can't find the container with id b06b37e325451ed9baedb6d566468d8b94f922d8a3e05d3476144cd7e9523bb0
Nov 02 01:23:23.963823 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:23.963789    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" event={"ID":"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd","Type":"ContainerStarted","Data":"b06b37e325451ed9baedb6d566468d8b94f922d8a3e05d3476144cd7e9523bb0"}
Nov 02 01:23:37.061945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:37.061903    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:23:37.062224 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:37.062007    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:23:37.062224 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:37.062049    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:23:38.025749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.025715    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" event={"ID":"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd","Type":"ContainerStarted","Data":"94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1"}
Nov 02 01:23:38.057835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.057279    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podStartSLOduration=1.765502514 podStartE2EDuration="15.057256961s" totalImagesPullingTime="13.291754447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:38.055097417 +0000 UTC m=+523.360030247" watchObservedRunningTime="2026-11-02 01:23:38.057256961 +0000 UTC m=+523.362189770"
Nov 02 01:23:38.184237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.184202    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"]
Nov 02 01:23:38.190361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.190342    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.203703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.203668    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"]
Nov 02 01:23:38.218430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.218402    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"]
Nov 02 01:23:38.222388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.222362    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.264751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.264706    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"]
Nov 02 01:23:38.312022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.311919    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.312022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.311967    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.312229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312054    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.312229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312081    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.312229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312152    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.312229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312188    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de30efc-e5bc-428c-ba8d-6b7967e29473-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.312229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312219    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84bn4\" (UniqueName: \"kubernetes.io/projected/0de30efc-e5bc-428c-ba8d-6b7967e29473-kube-api-access-84bn4\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.312529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312372    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1b737b-620b-4615-b94f-0ae145b61646-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.312529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312406    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.312529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312440    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.312529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312480    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.312529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312524    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1b737b-620b-4615-b94f-0ae145b61646-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.312732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312556    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9gf\" (UniqueName: \"kubernetes.io/projected/ee1b737b-620b-4615-b94f-0ae145b61646-kube-api-access-2r9gf\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.312732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312606    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1b737b-620b-4615-b94f-0ae145b61646-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.312732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312624    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.312732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312647    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.312732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312667    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.312732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.312683    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.413555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.413519    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1b737b-620b-4615-b94f-0ae145b61646-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.413555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.413553    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.413796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.413573    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.413796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.413589    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.413796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.413719    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1b737b-620b-4615-b94f-0ae145b61646-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.413796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.413775    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9gf\" (UniqueName: \"kubernetes.io/projected/ee1b737b-620b-4615-b94f-0ae145b61646-kube-api-access-2r9gf\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.413994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.413816    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1b737b-620b-4615-b94f-0ae145b61646-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.413994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.413843    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.413994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.413881    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.414166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414004    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.414166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414017    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.414166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414159    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.414363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414182    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.414363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414197    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.414363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414255    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.414363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414279    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.414583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414365    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.414583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414390    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.414583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414433    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.414583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414461    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de30efc-e5bc-428c-ba8d-6b7967e29473-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.414583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414500    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84bn4\" (UniqueName: \"kubernetes.io/projected/0de30efc-e5bc-428c-ba8d-6b7967e29473-kube-api-access-84bn4\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.414583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414570    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1b737b-620b-4615-b94f-0ae145b61646-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.414892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414575    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.414892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414751    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.414892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414459    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.414892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414818    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.415114 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.414965    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.415597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.415570    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de30efc-e5bc-428c-ba8d-6b7967e29473-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.416663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.416512    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1b737b-620b-4615-b94f-0ae145b61646-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.416771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.416678    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.417177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.417150    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1b737b-620b-4615-b94f-0ae145b61646-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.417177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.417164    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.426064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.426041    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9gf\" (UniqueName: \"kubernetes.io/projected/ee1b737b-620b-4615-b94f-0ae145b61646-kube-api-access-2r9gf\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.426147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.426112    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84bn4\" (UniqueName: \"kubernetes.io/projected/0de30efc-e5bc-428c-ba8d-6b7967e29473-kube-api-access-84bn4\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.426225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.426203    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de30efc-e5bc-428c-ba8d-6b7967e29473-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-ns4lc\" (UID: \"0de30efc-e5bc-428c-ba8d-6b7967e29473\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.426407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.426392    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1b737b-620b-4615-b94f-0ae145b61646-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-44d86\" (UID: \"ee1b737b-620b-4615-b94f-0ae145b61646\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.485485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.485449    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:38.485683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.485511    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:38.487116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.487092    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7f78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:23:38.487226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.487139    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 01:23:38.508217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.508183    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:38.567461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.567379    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:38.820411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.820309    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"]
Nov 02 01:23:38.822507 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:23:38.822466    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1b737b_620b_4615_b94f_0ae145b61646.slice/crio-4ff66ebc1e6ea00394447f8ad476b79413ec89ffbaefaa72ed4d712af71e60c9 WatchSource:0}: Error finding container 4ff66ebc1e6ea00394447f8ad476b79413ec89ffbaefaa72ed4d712af71e60c9: Status 404 returned error can't find the container with id 4ff66ebc1e6ea00394447f8ad476b79413ec89ffbaefaa72ed4d712af71e60c9
Nov 02 01:23:38.825803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.825770    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:23:38.825909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.825839    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:23:38.825909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.825869    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:23:38.882890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.882868    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"]
Nov 02 01:23:38.885491 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:23:38.885461    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de30efc_e5bc_428c_ba8d_6b7967e29473.slice/crio-739c547fc3771e3898800f4646a1e9702474119bf2e73809f125396387354241 WatchSource:0}: Error finding container 739c547fc3771e3898800f4646a1e9702474119bf2e73809f125396387354241: Status 404 returned error can't find the container with id 739c547fc3771e3898800f4646a1e9702474119bf2e73809f125396387354241
Nov 02 01:23:38.889200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.889166    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:23:38.889320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.889236    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:23:38.889320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:38.889263    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:23:39.036543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.036500    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" event={"ID":"ee1b737b-620b-4615-b94f-0ae145b61646","Type":"ContainerStarted","Data":"9b9258e2ac69e022d8371bbfdbeb307b10eaedd6cc6100bf37b58fc148da65c4"}
Nov 02 01:23:39.036827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.036801    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" event={"ID":"ee1b737b-620b-4615-b94f-0ae145b61646","Type":"ContainerStarted","Data":"4ff66ebc1e6ea00394447f8ad476b79413ec89ffbaefaa72ed4d712af71e60c9"}
Nov 02 01:23:39.057469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.057434    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" event={"ID":"0de30efc-e5bc-428c-ba8d-6b7967e29473","Type":"ContainerStarted","Data":"08ab384dceed120c728ab9801859b780b8f5c76926befc53222b5c2aecea6b4d"}
Nov 02 01:23:39.057746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.057724    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" event={"ID":"0de30efc-e5bc-428c-ba8d-6b7967e29473","Type":"ContainerStarted","Data":"739c547fc3771e3898800f4646a1e9702474119bf2e73809f125396387354241"}
Nov 02 01:23:39.071877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.071768    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podStartSLOduration=1.071747492 podStartE2EDuration="1.071747492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:39.070405049 +0000 UTC m=+524.375337877" watchObservedRunningTime="2026-11-02 01:23:39.071747492 +0000 UTC m=+524.376680387"
Nov 02 01:23:39.097558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.097493    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podStartSLOduration=1.097474248 podStartE2EDuration="1.097474248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:39.095456483 +0000 UTC m=+524.400389310" watchObservedRunningTime="2026-11-02 01:23:39.097474248 +0000 UTC m=+524.402407074"
Nov 02 01:23:39.488316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.488041    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7f78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:23:39.488316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.488112    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 01:23:39.510692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.510646    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:39.512341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.512134    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:23:39.512341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.512185    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 02 01:23:39.569989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.569901    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:39.570618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.570304    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-ns4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 01:23:39.570618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:39.570353    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podUID="0de30efc-e5bc-428c-ba8d-6b7967e29473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 01:23:40.488319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:40.487394    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7f78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:23:40.488319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:40.487456    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 01:23:40.508688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:40.508657    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:23:40.508920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:40.508901    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 02 01:23:40.570316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:40.569762    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-ns4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 01:23:40.570316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:40.569839    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podUID="0de30efc-e5bc-428c-ba8d-6b7967e29473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 01:23:41.485777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:41.485734    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7f78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:23:41.486087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:41.486060    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 01:23:41.508714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:41.508679    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:23:41.509333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:41.509280    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 02 01:23:41.572317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:41.571736    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-ns4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 01:23:41.572317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:41.571799    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podUID="0de30efc-e5bc-428c-ba8d-6b7967e29473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 01:23:42.486334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:42.485805    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7f78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:23:42.486334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:42.485884    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 01:23:42.509197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:42.509138    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:23:42.509703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:42.509237    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 02 01:23:42.576320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:42.574967    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-ns4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 01:23:42.576320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:42.575042    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podUID="0de30efc-e5bc-428c-ba8d-6b7967e29473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 01:23:43.485779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:43.485739    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7f78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:23:43.486073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:43.486044    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 01:23:43.509501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:43.509468    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:23:43.509984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:43.509525    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 02 01:23:43.571857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:43.571821    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-ns4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 01:23:43.572246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:43.572128    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podUID="0de30efc-e5bc-428c-ba8d-6b7967e29473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 01:23:44.485735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:44.485694    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7f78h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:23:44.486041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:44.486013    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 01:23:44.509008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:44.508977    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:23:44.509240 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:44.509220    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 02 01:23:44.570148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:44.570111    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-ns4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 01:23:44.570803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:44.570756    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podUID="0de30efc-e5bc-428c-ba8d-6b7967e29473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 01:23:45.371274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:45.371235    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:45.509120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:45.509075    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:23:45.509364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:45.509148    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 02 01:23:45.567879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:45.567836    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-ns4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 01:23:45.568083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:45.567906    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podUID="0de30efc-e5bc-428c-ba8d-6b7967e29473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 01:23:45.877046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:45.877017    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:45.878759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:45.878739    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:23:46.509597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:46.509556    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:23:46.509815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:46.509624    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Nov 02 01:23:46.568454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:46.568418    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-ns4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 01:23:46.568656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:46.568469    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podUID="0de30efc-e5bc-428c-ba8d-6b7967e29473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 01:23:47.512992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:47.512961    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:47.513511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:47.513252    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:47.513804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:47.513789    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86"
Nov 02 01:23:47.573683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:47.573655    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:47.574088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:47.574065    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:47.575029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:47.574992    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc"
Nov 02 01:23:47.659095 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:23:47.659056    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:23:49.857090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:49.857049    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-88cfb896-566x7"]
Nov 02 01:23:49.863810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:49.863789    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88cfb896-566x7"
Nov 02 01:23:49.869877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:49.869851    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-88cfb896-566x7"]
Nov 02 01:23:49.974852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:49.974793    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpxsp\" (UniqueName: \"kubernetes.io/projected/7a98bfd7-c685-408e-b963-686f01e3d7ad-kube-api-access-qpxsp\") pod \"authorino-88cfb896-566x7\" (UID: \"7a98bfd7-c685-408e-b963-686f01e3d7ad\") " pod="kuadrant-system/authorino-88cfb896-566x7"
Nov 02 01:23:49.975037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:49.974943    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a98bfd7-c685-408e-b963-686f01e3d7ad-tls-cert\") pod \"authorino-88cfb896-566x7\" (UID: \"7a98bfd7-c685-408e-b963-686f01e3d7ad\") " pod="kuadrant-system/authorino-88cfb896-566x7"
Nov 02 01:23:50.076278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:50.076241    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a98bfd7-c685-408e-b963-686f01e3d7ad-tls-cert\") pod \"authorino-88cfb896-566x7\" (UID: \"7a98bfd7-c685-408e-b963-686f01e3d7ad\") " pod="kuadrant-system/authorino-88cfb896-566x7"
Nov 02 01:23:50.076487 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:50.076342    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpxsp\" (UniqueName: \"kubernetes.io/projected/7a98bfd7-c685-408e-b963-686f01e3d7ad-kube-api-access-qpxsp\") pod \"authorino-88cfb896-566x7\" (UID: \"7a98bfd7-c685-408e-b963-686f01e3d7ad\") " pod="kuadrant-system/authorino-88cfb896-566x7"
Nov 02 01:23:50.078903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:50.078884    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a98bfd7-c685-408e-b963-686f01e3d7ad-tls-cert\") pod \"authorino-88cfb896-566x7\" (UID: \"7a98bfd7-c685-408e-b963-686f01e3d7ad\") " pod="kuadrant-system/authorino-88cfb896-566x7"
Nov 02 01:23:50.089224 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:50.089202    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpxsp\" (UniqueName: \"kubernetes.io/projected/7a98bfd7-c685-408e-b963-686f01e3d7ad-kube-api-access-qpxsp\") pod \"authorino-88cfb896-566x7\" (UID: \"7a98bfd7-c685-408e-b963-686f01e3d7ad\") " pod="kuadrant-system/authorino-88cfb896-566x7"
Nov 02 01:23:50.177198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:50.177172    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88cfb896-566x7"
Nov 02 01:23:50.453425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:50.453400    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-88cfb896-566x7"]
Nov 02 01:23:50.455938 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:23:50.455913    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a98bfd7_c685_408e_b963_686f01e3d7ad.slice/crio-f529f4ebbb5c8a513653590ac3ef9dc3c495cdb2ee58adc1a40875fe17b89926 WatchSource:0}: Error finding container f529f4ebbb5c8a513653590ac3ef9dc3c495cdb2ee58adc1a40875fe17b89926: Status 404 returned error can't find the container with id f529f4ebbb5c8a513653590ac3ef9dc3c495cdb2ee58adc1a40875fe17b89926
Nov 02 01:23:51.143124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:51.143077    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88cfb896-566x7" event={"ID":"7a98bfd7-c685-408e-b963-686f01e3d7ad","Type":"ContainerStarted","Data":"f529f4ebbb5c8a513653590ac3ef9dc3c495cdb2ee58adc1a40875fe17b89926"}
Nov 02 01:23:52.190580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:52.190533    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88cfb896-566x7" event={"ID":"7a98bfd7-c685-408e-b963-686f01e3d7ad","Type":"ContainerStarted","Data":"54bd621662a228547ee4370c0069498766594221b3c5f0de407b01362bcf162d"}
Nov 02 01:23:52.258314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:52.235999    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-88cfb896-566x7" podStartSLOduration=2.083646729 podStartE2EDuration="3.235977769s" totalImagesPullingTime="1.15233104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:52.214745496 +0000 UTC m=+537.519678325" watchObservedRunningTime="2026-11-02 01:23:52.235977769 +0000 UTC m=+537.540910596"
Nov 02 01:23:52.293365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:52.260225    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b5bd4d575-bjcs6"]
Nov 02 01:23:52.311395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:52.307548    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b5bd4d575-bjcs6" podUID="3f57c479-0991-432b-ac66-c4cb9c71d16f" containerName="authorino" containerID="cri-o://5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3" gracePeriod=30
Nov 02 01:23:52.818221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:52.816759    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:52.931538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:52.931513    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b5bd4d575-bjcs6"
Nov 02 01:23:53.124449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.124351    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f57c479-0991-432b-ac66-c4cb9c71d16f-tls-cert\" (UniqueName: \"kubernetes.io/secret/3f57c479-0991-432b-ac66-c4cb9c71d16f-tls-cert\") pod \"3f57c479-0991-432b-ac66-c4cb9c71d16f\" (UID: \"3f57c479-0991-432b-ac66-c4cb9c71d16f\") "
Nov 02 01:23:53.124628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.124473    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f57c479-0991-432b-ac66-c4cb9c71d16f-kube-api-access-wmbgm\" (UniqueName: \"kubernetes.io/projected/3f57c479-0991-432b-ac66-c4cb9c71d16f-kube-api-access-wmbgm\") pod \"3f57c479-0991-432b-ac66-c4cb9c71d16f\" (UID: \"3f57c479-0991-432b-ac66-c4cb9c71d16f\") "
Nov 02 01:23:53.126657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.126627    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f57c479-0991-432b-ac66-c4cb9c71d16f-kube-api-access-wmbgm" pod "3f57c479-0991-432b-ac66-c4cb9c71d16f" (UID: "3f57c479-0991-432b-ac66-c4cb9c71d16f"). InnerVolumeSpecName "kube-api-access-wmbgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:23:53.179728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.179691    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f57c479-0991-432b-ac66-c4cb9c71d16f-tls-cert" pod "3f57c479-0991-432b-ac66-c4cb9c71d16f" (UID: "3f57c479-0991-432b-ac66-c4cb9c71d16f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:23:53.208947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.208904    4851 generic.go:356] "Generic (PLEG): container finished" podID="3f57c479-0991-432b-ac66-c4cb9c71d16f" containerID="5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3" exitCode=0
Nov 02 01:23:53.209722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.209700    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b5bd4d575-bjcs6"
Nov 02 01:23:53.215311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.214364    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b5bd4d575-bjcs6" event={"ID":"3f57c479-0991-432b-ac66-c4cb9c71d16f","Type":"ContainerDied","Data":"5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3"}
Nov 02 01:23:53.215311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.214419    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b5bd4d575-bjcs6" event={"ID":"3f57c479-0991-432b-ac66-c4cb9c71d16f","Type":"ContainerDied","Data":"5564bd39465e325d7dfbd75d1c97f418631648ce286336ebee6b8224da3e02de"}
Nov 02 01:23:53.215311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.214442    4851 scope.go:122] "RemoveContainer" containerID="5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3"
Nov 02 01:23:53.225703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.225679    4851 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f57c479-0991-432b-ac66-c4cb9c71d16f-tls-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:23:53.225703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.225703    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmbgm\" (UniqueName: \"kubernetes.io/projected/3f57c479-0991-432b-ac66-c4cb9c71d16f-kube-api-access-wmbgm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:23:53.228848 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.228830    4851 scope.go:122] "RemoveContainer" containerID="5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3"
Nov 02 01:23:53.229237 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:23:53.229220    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3\": container with ID starting with 5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3 not found: ID does not exist" containerID="5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3"
Nov 02 01:23:53.229317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.229251    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3"} err="failed to get container status \"5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3\": rpc error: code = NotFound desc = could not find container \"5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3\": container with ID starting with 5b9bfd11bb319de3d3b75a740830f1e92ee1c751b7d47c80a4b3c337e360c5a3 not found: ID does not exist"
Nov 02 01:23:53.317317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.316032    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b5bd4d575-bjcs6"]
Nov 02 01:23:53.318700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.318674    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b5bd4d575-bjcs6"]
Nov 02 01:23:53.505315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:53.504284    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:54.846646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:54.846613    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f57c479-0991-432b-ac66-c4cb9c71d16f" path="/var/lib/kubelet/pods/3f57c479-0991-432b-ac66-c4cb9c71d16f/volumes"
Nov 02 01:23:58.863779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:58.863731    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:58.906100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:58.906056    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"]
Nov 02 01:23:58.906467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:58.906451    4851 cpu_manager.go:414] "containerMap: removing container" podUID="3f57c479-0991-432b-ac66-c4cb9c71d16f" containerName="authorino"
Nov 02 01:23:58.906467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:58.906465    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f57c479-0991-432b-ac66-c4cb9c71d16f" containerName="authorino"
Nov 02 01:23:58.906619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:58.906565    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f57c479-0991-432b-ac66-c4cb9c71d16f" containerName="authorino"
Nov 02 01:23:58.910993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:58.910973    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"
Nov 02 01:23:58.933392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:58.933354    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 01:23:58.934687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:58.934662    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bbqgh\""
Nov 02 01:23:58.936364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:58.936342    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"]
Nov 02 01:23:59.006314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:59.000805    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48plk\" (UniqueName: \"kubernetes.io/projected/b5d69b63-2892-44fd-a2ba-777e1278ecc5-kube-api-access-48plk\") pod \"payload-pre-processing-7b98bb678c-9vj42\" (UID: \"b5d69b63-2892-44fd-a2ba-777e1278ecc5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"
Nov 02 01:23:59.006314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:59.000881    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d69b63-2892-44fd-a2ba-777e1278ecc5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9vj42\" (UID: \"b5d69b63-2892-44fd-a2ba-777e1278ecc5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"
Nov 02 01:23:59.029316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:59.026217    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:23:59.101959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:59.101928    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48plk\" (UniqueName: \"kubernetes.io/projected/b5d69b63-2892-44fd-a2ba-777e1278ecc5-kube-api-access-48plk\") pod \"payload-pre-processing-7b98bb678c-9vj42\" (UID: \"b5d69b63-2892-44fd-a2ba-777e1278ecc5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"
Nov 02 01:23:59.102154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:59.101974    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d69b63-2892-44fd-a2ba-777e1278ecc5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9vj42\" (UID: \"b5d69b63-2892-44fd-a2ba-777e1278ecc5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"
Nov 02 01:23:59.102753 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:59.102729    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d69b63-2892-44fd-a2ba-777e1278ecc5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9vj42\" (UID: \"b5d69b63-2892-44fd-a2ba-777e1278ecc5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"
Nov 02 01:23:59.114024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:59.113955    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48plk\" (UniqueName: \"kubernetes.io/projected/b5d69b63-2892-44fd-a2ba-777e1278ecc5-kube-api-access-48plk\") pod \"payload-pre-processing-7b98bb678c-9vj42\" (UID: \"b5d69b63-2892-44fd-a2ba-777e1278ecc5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"
Nov 02 01:23:59.223516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:59.223478    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"
Nov 02 01:23:59.500244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:23:59.500213    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"]
Nov 02 01:23:59.503570 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:23:59.503542    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d69b63_2892_44fd_a2ba_777e1278ecc5.slice/crio-8ad6abba711117adb2f847b4d7dd00aead23c8568db363c6da50c57c91bd9a1f WatchSource:0}: Error finding container 8ad6abba711117adb2f847b4d7dd00aead23c8568db363c6da50c57c91bd9a1f: Status 404 returned error can't find the container with id 8ad6abba711117adb2f847b4d7dd00aead23c8568db363c6da50c57c91bd9a1f
Nov 02 01:24:00.245795 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:00.245766    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42" event={"ID":"b5d69b63-2892-44fd-a2ba-777e1278ecc5","Type":"ContainerStarted","Data":"8ad6abba711117adb2f847b4d7dd00aead23c8568db363c6da50c57c91bd9a1f"}
Nov 02 01:24:02.256458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:02.256362    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42" event={"ID":"b5d69b63-2892-44fd-a2ba-777e1278ecc5","Type":"ContainerStarted","Data":"45a4ef478f57353794e5d50fdc79bdaec76027730a535f46103b7cbc2261dec2"}
Nov 02 01:24:02.256458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:02.256457    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"
Nov 02 01:24:02.275393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:02.275348    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42" podStartSLOduration=1.789196487 podStartE2EDuration="4.275335827s" totalImagesPullingTime="2.48613934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:02.273552498 +0000 UTC m=+547.578485323" watchObservedRunningTime="2026-11-02 01:24:02.275335827 +0000 UTC m=+547.580268648"
Nov 02 01:24:08.860032 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:08.859998    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-8445457bbd-cjjmh"]
Nov 02 01:24:08.863360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:08.863344    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:24:08.867878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:08.867853    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 01:24:08.872320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:08.872273    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 01:24:08.872320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:08.872313    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-cb6fd\""
Nov 02 01:24:08.873825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:08.873805    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8445457bbd-cjjmh"]
Nov 02 01:24:08.969092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:08.969060    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 01:24:08.969092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:08.969069    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 01:24:08.998559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:08.998534    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4d91d9-ce2f-48f6-a164-720a5efcab51-metrics-certs\") pod \"ai-gateway-controller-8445457bbd-cjjmh\" (UID: \"fa4d91d9-ce2f-48f6-a164-720a5efcab51\") " pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:24:08.998661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:08.998562    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5g7f\" (UniqueName: \"kubernetes.io/projected/fa4d91d9-ce2f-48f6-a164-720a5efcab51-kube-api-access-m5g7f\") pod \"ai-gateway-controller-8445457bbd-cjjmh\" (UID: \"fa4d91d9-ce2f-48f6-a164-720a5efcab51\") " pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:24:09.099916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:09.099881    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4d91d9-ce2f-48f6-a164-720a5efcab51-metrics-certs\") pod \"ai-gateway-controller-8445457bbd-cjjmh\" (UID: \"fa4d91d9-ce2f-48f6-a164-720a5efcab51\") " pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:24:09.099916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:09.099914    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5g7f\" (UniqueName: \"kubernetes.io/projected/fa4d91d9-ce2f-48f6-a164-720a5efcab51-kube-api-access-m5g7f\") pod \"ai-gateway-controller-8445457bbd-cjjmh\" (UID: \"fa4d91d9-ce2f-48f6-a164-720a5efcab51\") " pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:24:09.102474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:09.102448    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4d91d9-ce2f-48f6-a164-720a5efcab51-metrics-certs\") pod \"ai-gateway-controller-8445457bbd-cjjmh\" (UID: \"fa4d91d9-ce2f-48f6-a164-720a5efcab51\") " pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:24:09.112780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:09.112732    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5g7f\" (UniqueName: \"kubernetes.io/projected/fa4d91d9-ce2f-48f6-a164-720a5efcab51-kube-api-access-m5g7f\") pod \"ai-gateway-controller-8445457bbd-cjjmh\" (UID: \"fa4d91d9-ce2f-48f6-a164-720a5efcab51\") " pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:24:09.176811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:09.176783    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:24:09.454601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:09.454576    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8445457bbd-cjjmh"]
Nov 02 01:24:09.456757 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:24:09.456725    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4d91d9_ce2f_48f6_a164_720a5efcab51.slice/crio-6a4442972b4e0c9cd30bcf74ffe77d123c31636043f9f5331e1af5f5d9182e72 WatchSource:0}: Error finding container 6a4442972b4e0c9cd30bcf74ffe77d123c31636043f9f5331e1af5f5d9182e72: Status 404 returned error can't find the container with id 6a4442972b4e0c9cd30bcf74ffe77d123c31636043f9f5331e1af5f5d9182e72
Nov 02 01:24:10.292814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:10.292772    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh" event={"ID":"fa4d91d9-ce2f-48f6-a164-720a5efcab51","Type":"ContainerStarted","Data":"6a4442972b4e0c9cd30bcf74ffe77d123c31636043f9f5331e1af5f5d9182e72"}
Nov 02 01:24:12.303738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:12.303646    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh" event={"ID":"fa4d91d9-ce2f-48f6-a164-720a5efcab51","Type":"ContainerStarted","Data":"08629e704e0ea18fb89e80705fff2b6078e21fde4947a6b5093ee1b5207eb8bd"}
Nov 02 01:24:12.304105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:12.303776    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:24:12.323159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:12.323106    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh" podStartSLOduration=1.740591295 podStartE2EDuration="4.323092643s" totalImagesPullingTime="2.582501348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:12.321607148 +0000 UTC m=+557.626540022" watchObservedRunningTime="2026-11-02 01:24:12.323092643 +0000 UTC m=+557.628025506"
Nov 02 01:24:13.263754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:13.263724    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42"
Nov 02 01:24:17.653781 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:24:17.653747    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:24:23.310966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:23.310937    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:24:25.135086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:25.135050    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:25.666761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:25.666730    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:27.387495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:27.387461    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:27.523858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:27.523825    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:29.048554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:29.048523    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:29.187558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:29.187522    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:37.270042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:37.269964    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:37.403576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:37.403545    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:45.436943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:45.436908    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:24:47.654492 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:24:47.654459    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:24:54.893515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:54.893487    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:24:54.896836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:24:54.896816    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:25:08.045591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.045556    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:08.189962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.189932    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:08.550904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.550502    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"]
Nov 02 01:25:08.554593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.554569    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.563412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.563392    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-b9d816-openshift-default-dockercfg-hvdvt\""
Nov 02 01:25:08.578149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.578105    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"]
Nov 02 01:25:08.614188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.614160    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.614359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.614192    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.614359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.614215    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.614359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.614232    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.614474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.614411    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.614474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.614440    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqvbq\" (UniqueName: \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-kube-api-access-bqvbq\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.614537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.614481    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e45f01-7368-4448-bbec-2557dd42e004-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.614537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.614516    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e45f01-7368-4448-bbec-2557dd42e004-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.614599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.614580    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.642375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.642327    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:08.653698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.653665    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"]
Nov 02 01:25:08.657191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.657170    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.664382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.664326    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5687bc-openshift-default-dockercfg-mw8fh\""
Nov 02 01:25:08.670381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.670356    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"]
Nov 02 01:25:08.715685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.715639    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.715685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.715672    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.715685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.715691    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqvbq\" (UniqueName: \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-kube-api-access-bqvbq\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.716029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.715733    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.716029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.715757    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.716029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.715782    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxh6w\" (UniqueName: \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-kube-api-access-pxh6w\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.716029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.715811    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.716029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.715830    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.716029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.715935    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.716029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.716004    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.716411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.716238    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e45f01-7368-4448-bbec-2557dd42e004-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.716411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.716275    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e45f01-7368-4448-bbec-2557dd42e004-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.716411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.716336    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.716411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.716350    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.716411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.716372    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.716958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.716417    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.716958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.716447    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.716958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.716780    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.716958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.716816    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.717173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.717083    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e45f01-7368-4448-bbec-2557dd42e004-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.717259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.717220    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.717367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.717263    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.717367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.717352    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.718557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.718538    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.719506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.719484    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e45f01-7368-4448-bbec-2557dd42e004-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.730166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.730141    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.730732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.730710    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqvbq\" (UniqueName: \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-kube-api-access-bqvbq\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.787377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.787345    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:08.818021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.817939    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.817972    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.817991    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.818061    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.818094    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.818134    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxh6w\" (UniqueName: \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-kube-api-access-pxh6w\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.818179    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.818205    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.818232    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.818407    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.818477    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.818904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.818876    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.819178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.819009    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.819307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.819255    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.821079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.820874    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.821220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.821195    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.824700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.824681    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"]
Nov 02 01:25:08.829818 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.829799    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:08.832762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.832741    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.837702 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.837680    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxh6w\" (UniqueName: \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-kube-api-access-pxh6w\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.846801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.846777    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"]
Nov 02 01:25:08.859629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.859601    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b6ec07-openshift-default-dockercfg-9l7xh\""
Nov 02 01:25:08.870471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.870447    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:08.919893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.919705    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:08.920089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.919971    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:08.920089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.920006    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:08.920214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.920149    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:08.920214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.920185    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf2z2\" (UniqueName: \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-kube-api-access-tf2z2\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:08.921538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.920228    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:08.921538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.920271    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:08.921538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.920311    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:08.921538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.920341    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:08.938024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.937994    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:08.971039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.970692    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:08.999621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:08.999591    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"]
Nov 02 01:25:09.004834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.004807    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.012273 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.012157    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5ed42b-openshift-default-dockercfg-6p624\""
Nov 02 01:25:09.018229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.018155    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"]
Nov 02 01:25:09.021186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.021160    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.021414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.021395    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.023469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.023445    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.024062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.024026    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.024215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.024199    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.024996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.024344    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.024996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.024429    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.024996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.024458    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf2z2\" (UniqueName: \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-kube-api-access-tf2z2\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.024996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.024462    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.024996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.024497    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.024996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.024547    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.025455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.025434    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.025521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.025476    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.026005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.025983    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.026580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.026552    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.027632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.027611    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.042391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.042337    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.048269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.048234    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf2z2\" (UniqueName: \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-kube-api-access-tf2z2\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.089560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.089485    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:09.125713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.125686    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.125874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.125718    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.125874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.125749    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.125874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.125822    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.126033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.125923    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.126033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.125951    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.126033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.125983    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.126138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.126053    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdg54\" (UniqueName: \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-kube-api-access-gdg54\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.126138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.126084    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.160461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.160433    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:09.169529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.169503    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"]
Nov 02 01:25:09.172965 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:09.172939    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e45f01_7368_4448_bbec_2557dd42e004.slice/crio-6da843f7868e0e19c76b2cbfb855f7bd2a741bcc83c2609314199808576f093b WatchSource:0}: Error finding container 6da843f7868e0e19c76b2cbfb855f7bd2a741bcc83c2609314199808576f093b: Status 404 returned error can't find the container with id 6da843f7868e0e19c76b2cbfb855f7bd2a741bcc83c2609314199808576f093b
Nov 02 01:25:09.175600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.175533    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.175724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.175704    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.175790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.175750    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.224777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.224689    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:09.229746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.229686    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.230484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.230268    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.230602    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.230655    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.230685    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdg54\" (UniqueName: \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-kube-api-access-gdg54\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.230709    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.230744    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.230761    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.230792    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.230815    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.231068    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.231545    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.231988    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.232737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.232045    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.234108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.234062    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.234209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.234133    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.243820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.243797    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.243912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.243896    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdg54\" (UniqueName: \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-kube-api-access-gdg54\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.261960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.261935    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"]
Nov 02 01:25:09.266324 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:09.266277    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3849006_d06b_4c97_840d_5f34aa6cdfe1.slice/crio-350cd280e339afb35935557c5bfe264e59d477d3aad50dfba04258a510eab79b WatchSource:0}: Error finding container 350cd280e339afb35935557c5bfe264e59d477d3aad50dfba04258a510eab79b: Status 404 returned error can't find the container with id 350cd280e339afb35935557c5bfe264e59d477d3aad50dfba04258a510eab79b
Nov 02 01:25:09.269343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.269280    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.269439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.269371    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.269439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.269402    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.334731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.334674    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:09.367996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.367879    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:09.456567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.456133    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"]
Nov 02 01:25:09.468313 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:09.468251    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4ce5ae_b0b2_4ba7_8c04_672afae643ed.slice/crio-f4491f6fb7c5ea164fcf763e0a52816df80ff659de4a7a818cbb370f31262ba5 WatchSource:0}: Error finding container f4491f6fb7c5ea164fcf763e0a52816df80ff659de4a7a818cbb370f31262ba5: Status 404 returned error can't find the container with id f4491f6fb7c5ea164fcf763e0a52816df80ff659de4a7a818cbb370f31262ba5
Nov 02 01:25:09.480981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.480937    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.481115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.481028    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.481115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.481099    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.517906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.517843    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:09.562362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.562324    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" event={"ID":"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed","Type":"ContainerStarted","Data":"f4491f6fb7c5ea164fcf763e0a52816df80ff659de4a7a818cbb370f31262ba5"}
Nov 02 01:25:09.565329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.564887    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" event={"ID":"f8e45f01-7368-4448-bbec-2557dd42e004","Type":"ContainerStarted","Data":"2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5"}
Nov 02 01:25:09.565329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.564940    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" event={"ID":"f8e45f01-7368-4448-bbec-2557dd42e004","Type":"ContainerStarted","Data":"6da843f7868e0e19c76b2cbfb855f7bd2a741bcc83c2609314199808576f093b"}
Nov 02 01:25:09.566808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.566784    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" event={"ID":"a3849006-d06b-4c97-840d-5f34aa6cdfe1","Type":"ContainerStarted","Data":"31b2bdd4e82af455230be053b83d6fbaed4cb1d93516a41beb780de730fcbeba"}
Nov 02 01:25:09.566903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.566840    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" event={"ID":"a3849006-d06b-4c97-840d-5f34aa6cdfe1","Type":"ContainerStarted","Data":"350cd280e339afb35935557c5bfe264e59d477d3aad50dfba04258a510eab79b"}
Nov 02 01:25:09.588576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.588518    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podStartSLOduration=1.588498537 podStartE2EDuration="1.588498537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:09.587140605 +0000 UTC m=+614.892073442" watchObservedRunningTime="2026-11-02 01:25:09.588498537 +0000 UTC m=+614.893431365"
Nov 02 01:25:09.619806 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.619703    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podStartSLOduration=1.619682692 podStartE2EDuration="1.619682692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:09.617585787 +0000 UTC m=+614.922518613" watchObservedRunningTime="2026-11-02 01:25:09.619682692 +0000 UTC m=+614.924615519"
Nov 02 01:25:09.665420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.665393    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"]
Nov 02 01:25:09.667533 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:09.667498    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c39b84_1325_4b9f_8cbf_ef64ecf31237.slice/crio-97a0f30d0b4c7882b9d467fd49b8df59c69c5f3eac036d06ae5dbba794298889 WatchSource:0}: Error finding container 97a0f30d0b4c7882b9d467fd49b8df59c69c5f3eac036d06ae5dbba794298889: Status 404 returned error can't find the container with id 97a0f30d0b4c7882b9d467fd49b8df59c69c5f3eac036d06ae5dbba794298889
Nov 02 01:25:09.670353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.670320    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.670454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.670388    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.670454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.670423    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:09.871665 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.871534    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:09.971111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:09.971083    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:10.573465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:10.573427    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" event={"ID":"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed","Type":"ContainerStarted","Data":"940927f8e46c5f52d34abf6b77bcd7d1e975c6dbf3c193eb2a358f3507425244"}
Nov 02 01:25:10.575305 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:10.575268    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" event={"ID":"93c39b84-1325-4b9f-8cbf-ef64ecf31237","Type":"ContainerStarted","Data":"b68d48cf98a0c88d1052781961474a3cda98e2a273a98fce62672a7a0f898d1a"}
Nov 02 01:25:10.575440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:10.575348    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" event={"ID":"93c39b84-1325-4b9f-8cbf-ef64ecf31237","Type":"ContainerStarted","Data":"97a0f30d0b4c7882b9d467fd49b8df59c69c5f3eac036d06ae5dbba794298889"}
Nov 02 01:25:10.597072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:10.597026    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" podStartSLOduration=2.59701426 podStartE2EDuration="2.59701426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:10.595685316 +0000 UTC m=+615.900618143" watchObservedRunningTime="2026-11-02 01:25:10.59701426 +0000 UTC m=+615.901947086"
Nov 02 01:25:10.623125 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:10.623071    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" podStartSLOduration=2.623054812 podStartE2EDuration="2.623054812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:10.620704838 +0000 UTC m=+615.925637666" watchObservedRunningTime="2026-11-02 01:25:10.623054812 +0000 UTC m=+615.927987641"
Nov 02 01:25:10.876331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:10.876239    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:10.977371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:10.977342    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:11.161041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.161010    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:11.166775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.166751    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:11.336233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.336197    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:11.341581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.341554    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:11.581434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.581007    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:11.581434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.581063    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:11.581434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.581088    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:11.581434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.581110    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:11.582030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.581845    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:11.582075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.582034    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:11.582075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.582071    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:11.582130 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:11.582101    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:12.014816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:12.014782    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:12.151129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:12.151093    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:12.274894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:12.274811    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:12.625515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:12.625432    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:12.766970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:12.766935    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:12.904489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:12.904398    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:13.081241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:13.081207    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:13.311127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:13.311090    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:13.447269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:13.447239    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:13.603980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:13.603902    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:14.253318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:14.253259    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:14.404203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:14.404171    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:14.713775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:14.713746    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:14.856163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:14.856133    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:14.989047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:14.988971    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:15.221721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.221689    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"]
Nov 02 01:25:15.226370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.226346    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:25:15.230052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.230031    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-123986\""
Nov 02 01:25:15.234810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.234783    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:15.239586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.239512    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"]
Nov 02 01:25:15.312076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.312040    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssbh\" (UniqueName: \"kubernetes.io/projected/1e34ceb8-6e96-4404-bfd7-728e16738621-kube-api-access-bssbh\") pod \"payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st\" (UID: \"1e34ceb8-6e96-4404-bfd7-728e16738621\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:25:15.312493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.312156    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e34ceb8-6e96-4404-bfd7-728e16738621-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st\" (UID: \"1e34ceb8-6e96-4404-bfd7-728e16738621\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:25:15.387315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.377273    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:15.414021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.413981    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssbh\" (UniqueName: \"kubernetes.io/projected/1e34ceb8-6e96-4404-bfd7-728e16738621-kube-api-access-bssbh\") pod \"payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st\" (UID: \"1e34ceb8-6e96-4404-bfd7-728e16738621\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:25:15.414206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.414114    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e34ceb8-6e96-4404-bfd7-728e16738621-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st\" (UID: \"1e34ceb8-6e96-4404-bfd7-728e16738621\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:25:15.415093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.415067    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e34ceb8-6e96-4404-bfd7-728e16738621-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st\" (UID: \"1e34ceb8-6e96-4404-bfd7-728e16738621\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:25:15.428285 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.428256    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssbh\" (UniqueName: \"kubernetes.io/projected/1e34ceb8-6e96-4404-bfd7-728e16738621-kube-api-access-bssbh\") pod \"payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st\" (UID: \"1e34ceb8-6e96-4404-bfd7-728e16738621\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:25:15.543369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.543259    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:25:15.650693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.650660    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:15.690637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.690607    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"]
Nov 02 01:25:15.694225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.694209    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:25:15.706945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.706920    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"]
Nov 02 01:25:15.707064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.707051    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-47dd19\""
Nov 02 01:25:15.727149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.727122    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"]
Nov 02 01:25:15.730582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.730552    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:25:15.737851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.737801    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"]
Nov 02 01:25:15.738817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.738795    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-47dd19-dockercfg-zft89\""
Nov 02 01:25:15.786678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.786649    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:15.825361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.821837    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvl7\" (UniqueName: \"kubernetes.io/projected/14b4b0ad-6f18-410d-b545-db6f1ecd3141-kube-api-access-4nvl7\") pod \"payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p\" (UID: \"14b4b0ad-6f18-410d-b545-db6f1ecd3141\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:25:15.825361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.821943    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whcwf\" (UniqueName: \"kubernetes.io/projected/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-kube-api-access-whcwf\") pod \"payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h\" (UID: \"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:25:15.825361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.822022    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b4b0ad-6f18-410d-b545-db6f1ecd3141-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p\" (UID: \"14b4b0ad-6f18-410d-b545-db6f1ecd3141\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:25:15.825361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.822092    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h\" (UID: \"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:25:15.858805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.858773    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"]
Nov 02 01:25:15.912044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.912016    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:15.923262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.923239    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b4b0ad-6f18-410d-b545-db6f1ecd3141-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p\" (UID: \"14b4b0ad-6f18-410d-b545-db6f1ecd3141\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:25:15.923382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.923338    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h\" (UID: \"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:25:15.923436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.923390    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvl7\" (UniqueName: \"kubernetes.io/projected/14b4b0ad-6f18-410d-b545-db6f1ecd3141-kube-api-access-4nvl7\") pod \"payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p\" (UID: \"14b4b0ad-6f18-410d-b545-db6f1ecd3141\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:25:15.923475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.923461    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whcwf\" (UniqueName: \"kubernetes.io/projected/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-kube-api-access-whcwf\") pod \"payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h\" (UID: \"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:25:15.924010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.923994    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b4b0ad-6f18-410d-b545-db6f1ecd3141-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p\" (UID: \"14b4b0ad-6f18-410d-b545-db6f1ecd3141\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:25:15.924010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.924004    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h\" (UID: \"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:25:15.939279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.939192    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvl7\" (UniqueName: \"kubernetes.io/projected/14b4b0ad-6f18-410d-b545-db6f1ecd3141-kube-api-access-4nvl7\") pod \"payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p\" (UID: \"14b4b0ad-6f18-410d-b545-db6f1ecd3141\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:25:15.940109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:15.940055    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whcwf\" (UniqueName: \"kubernetes.io/projected/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-kube-api-access-whcwf\") pod \"payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h\" (UID: \"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:25:16.009931 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.009902    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:25:16.045602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.045558    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:25:16.078029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.077964    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:16.089913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.089886    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"]
Nov 02 01:25:16.095158 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.095141    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:25:16.102927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.102852    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ca1c53\""
Nov 02 01:25:16.108256 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.108122    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"]
Nov 02 01:25:16.129174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.129147    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"]
Nov 02 01:25:16.149367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.149339    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"]
Nov 02 01:25:16.150092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.150061    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:25:16.155061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.155039    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ca1c53-dockercfg-xxvlz\""
Nov 02 01:25:16.211876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.211849    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:16.226367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.226338    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln\" (UID: \"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:25:16.226506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.226381    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bfz2\" (UniqueName: \"kubernetes.io/projected/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-kube-api-access-4bfz2\") pod \"payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln\" (UID: \"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:25:16.226572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.226542    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pvhk\" (UniqueName: \"kubernetes.io/projected/c6ac98d6-c879-4606-81a4-b791e4af5b3f-kube-api-access-6pvhk\") pod \"payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8\" (UID: \"c6ac98d6-c879-4606-81a4-b791e4af5b3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:25:16.226626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.226575    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ac98d6-c879-4606-81a4-b791e4af5b3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8\" (UID: \"c6ac98d6-c879-4606-81a4-b791e4af5b3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:25:16.291370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.291338    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"]
Nov 02 01:25:16.295198 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:16.295167    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b4b0ad_6f18_410d_b545_db6f1ecd3141.slice/crio-41c603172a7b06bd1af49163852eb9a38f8835c843f60c835fb07523f118df4e WatchSource:0}: Error finding container 41c603172a7b06bd1af49163852eb9a38f8835c843f60c835fb07523f118df4e: Status 404 returned error can't find the container with id 41c603172a7b06bd1af49163852eb9a38f8835c843f60c835fb07523f118df4e
Nov 02 01:25:16.332771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.327956    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pvhk\" (UniqueName: \"kubernetes.io/projected/c6ac98d6-c879-4606-81a4-b791e4af5b3f-kube-api-access-6pvhk\") pod \"payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8\" (UID: \"c6ac98d6-c879-4606-81a4-b791e4af5b3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:25:16.332771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.328023    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ac98d6-c879-4606-81a4-b791e4af5b3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8\" (UID: \"c6ac98d6-c879-4606-81a4-b791e4af5b3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:25:16.332771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.328144    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln\" (UID: \"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:25:16.332771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.328180    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bfz2\" (UniqueName: \"kubernetes.io/projected/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-kube-api-access-4bfz2\") pod \"payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln\" (UID: \"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:25:16.332771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.329561    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ac98d6-c879-4606-81a4-b791e4af5b3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8\" (UID: \"c6ac98d6-c879-4606-81a4-b791e4af5b3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:25:16.332771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.330389    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln\" (UID: \"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:25:16.357317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.352744    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bfz2\" (UniqueName: \"kubernetes.io/projected/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-kube-api-access-4bfz2\") pod \"payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln\" (UID: \"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:25:16.366977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.366942    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pvhk\" (UniqueName: \"kubernetes.io/projected/c6ac98d6-c879-4606-81a4-b791e4af5b3f-kube-api-access-6pvhk\") pod \"payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8\" (UID: \"c6ac98d6-c879-4606-81a4-b791e4af5b3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:25:16.402383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.402351    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"]
Nov 02 01:25:16.425389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.425365    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:25:16.465659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.465620    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:25:16.590928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.590839    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:16.610359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.610283    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st" event={"ID":"1e34ceb8-6e96-4404-bfd7-728e16738621","Type":"ContainerStarted","Data":"d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3"}
Nov 02 01:25:16.610521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.610379    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st" event={"ID":"1e34ceb8-6e96-4404-bfd7-728e16738621","Type":"ContainerStarted","Data":"b3660ae04552b2d965e77370bc89a7bc1fd3390637941143537528259dd994fa"}
Nov 02 01:25:16.610521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.610413    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:25:16.612129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.612101    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p" event={"ID":"14b4b0ad-6f18-410d-b545-db6f1ecd3141","Type":"ContainerStarted","Data":"e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba"}
Nov 02 01:25:16.612254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.612149    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p" event={"ID":"14b4b0ad-6f18-410d-b545-db6f1ecd3141","Type":"ContainerStarted","Data":"41c603172a7b06bd1af49163852eb9a38f8835c843f60c835fb07523f118df4e"}
Nov 02 01:25:16.612254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.612178    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:25:16.613636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.613616    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h" event={"ID":"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68","Type":"ContainerStarted","Data":"82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c"}
Nov 02 01:25:16.613750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.613647    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h" event={"ID":"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68","Type":"ContainerStarted","Data":"0a4fe3b27bc60aef3a4e99cb28d4d6c8f7f4b9b9c88e88ccf656add2af16919f"}
Nov 02 01:25:16.613792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.613752    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:25:16.635134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.635089    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st" podStartSLOduration=1.635061943 podStartE2EDuration="1.635061943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:16.633388393 +0000 UTC m=+621.938321216" watchObservedRunningTime="2026-11-02 01:25:16.635061943 +0000 UTC m=+621.939994769"
Nov 02 01:25:16.666120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.666056    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p" podStartSLOduration=1.666038492 podStartE2EDuration="1.666038492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:16.665487111 +0000 UTC m=+621.970419928" watchObservedRunningTime="2026-11-02 01:25:16.666038492 +0000 UTC m=+621.970971320"
Nov 02 01:25:16.690954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.690898    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h" podStartSLOduration=1.690884669 podStartE2EDuration="1.690884669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:16.690836441 +0000 UTC m=+621.995769248" watchObservedRunningTime="2026-11-02 01:25:16.690884669 +0000 UTC m=+621.995817499"
Nov 02 01:25:16.719370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.719345    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"]
Nov 02 01:25:16.721327 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:16.721177    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10e0dc1_0c5b_4bc6_bb28_06a129774bfe.slice/crio-be0ee497134abc716ea74df66cec3ded4b6ee39253f8f0394d268a88e1e44157 WatchSource:0}: Error finding container be0ee497134abc716ea74df66cec3ded4b6ee39253f8f0394d268a88e1e44157: Status 404 returned error can't find the container with id be0ee497134abc716ea74df66cec3ded4b6ee39253f8f0394d268a88e1e44157
Nov 02 01:25:16.725783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.725755    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:16.756797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.756778    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"]
Nov 02 01:25:16.759426 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:16.759396    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ac98d6_c879_4606_81a4_b791e4af5b3f.slice/crio-62fa6380c31f500f29b99e1a572cde9cf85f9ce3444d0eb85e2d46f55dc13085 WatchSource:0}: Error finding container 62fa6380c31f500f29b99e1a572cde9cf85f9ce3444d0eb85e2d46f55dc13085: Status 404 returned error can't find the container with id 62fa6380c31f500f29b99e1a572cde9cf85f9ce3444d0eb85e2d46f55dc13085
Nov 02 01:25:16.854180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:16.854091    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:17.311966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.311933    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"]
Nov 02 01:25:17.315684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.315667    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:25:17.323116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.323089    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:17.324099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.324076    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5ed42b\""
Nov 02 01:25:17.326341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.326316    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"]
Nov 02 01:25:17.443403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.443357    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjmv5\" (UniqueName: \"kubernetes.io/projected/1c6fddd1-829f-4aef-88d8-86ecab8099c6-kube-api-access-wjmv5\") pod \"payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw\" (UID: \"1c6fddd1-829f-4aef-88d8-86ecab8099c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:25:17.443933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.443549    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6fddd1-829f-4aef-88d8-86ecab8099c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw\" (UID: \"1c6fddd1-829f-4aef-88d8-86ecab8099c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:25:17.462621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.462417    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:17.544667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.544629    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmv5\" (UniqueName: \"kubernetes.io/projected/1c6fddd1-829f-4aef-88d8-86ecab8099c6-kube-api-access-wjmv5\") pod \"payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw\" (UID: \"1c6fddd1-829f-4aef-88d8-86ecab8099c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:25:17.544843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.544715    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6fddd1-829f-4aef-88d8-86ecab8099c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw\" (UID: \"1c6fddd1-829f-4aef-88d8-86ecab8099c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:25:17.545510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.545488    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6fddd1-829f-4aef-88d8-86ecab8099c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw\" (UID: \"1c6fddd1-829f-4aef-88d8-86ecab8099c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:25:17.560794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.560770    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmv5\" (UniqueName: \"kubernetes.io/projected/1c6fddd1-829f-4aef-88d8-86ecab8099c6-kube-api-access-wjmv5\") pod \"payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw\" (UID: \"1c6fddd1-829f-4aef-88d8-86ecab8099c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:25:17.620934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.620850    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8" event={"ID":"c6ac98d6-c879-4606-81a4-b791e4af5b3f","Type":"ContainerStarted","Data":"54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d"}
Nov 02 01:25:17.620934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.620900    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8" event={"ID":"c6ac98d6-c879-4606-81a4-b791e4af5b3f","Type":"ContainerStarted","Data":"62fa6380c31f500f29b99e1a572cde9cf85f9ce3444d0eb85e2d46f55dc13085"}
Nov 02 01:25:17.621136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.620967    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:25:17.622536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.622511    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln" event={"ID":"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe","Type":"ContainerStarted","Data":"281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b"}
Nov 02 01:25:17.622696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.622547    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln" event={"ID":"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe","Type":"ContainerStarted","Data":"be0ee497134abc716ea74df66cec3ded4b6ee39253f8f0394d268a88e1e44157"}
Nov 02 01:25:17.631687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.631666    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:25:17.639407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.639359    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8" podStartSLOduration=1.639344688 podStartE2EDuration="1.639344688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:17.638373231 +0000 UTC m=+622.943306059" watchObservedRunningTime="2026-11-02 01:25:17.639344688 +0000 UTC m=+622.944277515"
Nov 02 01:25:17.639790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.639749    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:17.660025 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:25:17.659975    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:25:17.662094 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.661071    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln" podStartSLOduration=1.66105339 podStartE2EDuration="1.66105339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:17.660151256 +0000 UTC m=+622.965084085" watchObservedRunningTime="2026-11-02 01:25:17.66105339 +0000 UTC m=+622.965986266"
Nov 02 01:25:17.793316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.790463    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:17.917045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:17.917021    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"]
Nov 02 01:25:17.918264 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:17.918243    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6fddd1_829f_4aef_88d8_86ecab8099c6.slice/crio-1187d4d4ea7e54e819c2be92d861170c129a1ad56c9c5b95bff88c6ba2f62e0e WatchSource:0}: Error finding container 1187d4d4ea7e54e819c2be92d861170c129a1ad56c9c5b95bff88c6ba2f62e0e: Status 404 returned error can't find the container with id 1187d4d4ea7e54e819c2be92d861170c129a1ad56c9c5b95bff88c6ba2f62e0e
Nov 02 01:25:18.037575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.037547    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:18.082764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.082731    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"]
Nov 02 01:25:18.086371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.086353    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:25:18.096611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.096582    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b6ec07-dockercfg-j5v8z\""
Nov 02 01:25:18.098760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.098732    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"]
Nov 02 01:25:18.098922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.098891    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b6ec07\""
Nov 02 01:25:18.152317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.151830    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41550ac8-635a-4162-bbce-4145804093be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc\" (UID: \"41550ac8-635a-4162-bbce-4145804093be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:25:18.152317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.152111    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb6pp\" (UniqueName: \"kubernetes.io/projected/41550ac8-635a-4162-bbce-4145804093be-kube-api-access-qb6pp\") pod \"payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc\" (UID: \"41550ac8-635a-4162-bbce-4145804093be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:25:18.184359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.184110    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:18.253384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.253340    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb6pp\" (UniqueName: \"kubernetes.io/projected/41550ac8-635a-4162-bbce-4145804093be-kube-api-access-qb6pp\") pod \"payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc\" (UID: \"41550ac8-635a-4162-bbce-4145804093be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:25:18.253575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.253456    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41550ac8-635a-4162-bbce-4145804093be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc\" (UID: \"41550ac8-635a-4162-bbce-4145804093be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:25:18.254189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.254166    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41550ac8-635a-4162-bbce-4145804093be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc\" (UID: \"41550ac8-635a-4162-bbce-4145804093be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:25:18.265603 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.265573    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb6pp\" (UniqueName: \"kubernetes.io/projected/41550ac8-635a-4162-bbce-4145804093be-kube-api-access-qb6pp\") pod \"payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc\" (UID: \"41550ac8-635a-4162-bbce-4145804093be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:25:18.403474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.403431    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:25:18.630996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.630948    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw" event={"ID":"1c6fddd1-829f-4aef-88d8-86ecab8099c6","Type":"ContainerStarted","Data":"f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772"}
Nov 02 01:25:18.631416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.631017    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw" event={"ID":"1c6fddd1-829f-4aef-88d8-86ecab8099c6","Type":"ContainerStarted","Data":"1187d4d4ea7e54e819c2be92d861170c129a1ad56c9c5b95bff88c6ba2f62e0e"}
Nov 02 01:25:18.631416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.631182    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:25:18.631507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.631448    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:25:18.650970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.650911    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw" podStartSLOduration=1.650893833 podStartE2EDuration="1.650893833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:18.648581977 +0000 UTC m=+623.953514814" watchObservedRunningTime="2026-11-02 01:25:18.650893833 +0000 UTC m=+623.955826661"
Nov 02 01:25:18.721428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.721396    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:18.729777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:18.729748    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"]
Nov 02 01:25:18.730447 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:18.730421    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41550ac8_635a_4162_bbce_4145804093be.slice/crio-f88ec6b9665156be67b43e98478b5c875725f83561285281a44b0588b0ea8217 WatchSource:0}: Error finding container f88ec6b9665156be67b43e98478b5c875725f83561285281a44b0588b0ea8217: Status 404 returned error can't find the container with id f88ec6b9665156be67b43e98478b5c875725f83561285281a44b0588b0ea8217
Nov 02 01:25:19.695140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:19.695082    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc" event={"ID":"41550ac8-635a-4162-bbce-4145804093be","Type":"ContainerStarted","Data":"715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c"}
Nov 02 01:25:19.695804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:19.695783    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:25:19.695887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:19.695848    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc" event={"ID":"41550ac8-635a-4162-bbce-4145804093be","Type":"ContainerStarted","Data":"f88ec6b9665156be67b43e98478b5c875725f83561285281a44b0588b0ea8217"}
Nov 02 01:25:19.715103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:19.715059    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc" podStartSLOduration=1.715046922 podStartE2EDuration="1.715046922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:19.713484253 +0000 UTC m=+625.018417079" watchObservedRunningTime="2026-11-02 01:25:19.715046922 +0000 UTC m=+625.019979742"
Nov 02 01:25:19.728472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:19.728449    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:19.861095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:19.861044    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:20.626627 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:20.626595    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:21.523415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:21.523385    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:21.650983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:21.650917    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:21.781373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:21.780845    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:22.420420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:22.420374    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:22.557408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:22.554151    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:23.364315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:23.363905    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:24.386589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:24.386539    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:25.585783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:25.585739    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:26.943133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:26.943060    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:27.626525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:27.626266    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:25:27.626525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:27.626431    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:25:27.626525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:27.626504    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:25:27.627466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:27.627437    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:25:27.627536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:27.627451    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:25:27.627536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:27.627499    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:25:27.627606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:27.627546    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:25:27.813733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:27.813697    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:27.959174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:27.956788    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:28.353496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:28.353333    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:28.492375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:28.492332    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:28.619642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:28.619569    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:28.655321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:28.643536    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:25:28.781421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:28.764816    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:29.299839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.299173    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"]
Nov 02 01:25:29.305930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.305904    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.321882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.321660    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47dd19\"/\"simulator-w2-47dd19-kserve-self-signed-certs\""
Nov 02 01:25:29.321882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.321682    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47dd19\"/\"default-dockercfg-4qqtf\""
Nov 02 01:25:29.321882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.321810    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47dd19\"/\"kube-root-ca.crt\""
Nov 02 01:25:29.321882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.321856    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47dd19\"/\"openshift-service-ca.crt\""
Nov 02 01:25:29.325356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.325311    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"]
Nov 02 01:25:29.385533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.376175    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:29.401091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.401051    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-home\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.401269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.401130    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhz8p\" (UniqueName: \"kubernetes.io/projected/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-kube-api-access-qhz8p\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.401269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.401174    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tls-certs\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.401269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.401242    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-model-cache\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.401482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.401284    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-dshm\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.401482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.401355    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tmp-dir\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.512889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.510923    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-home\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.512889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.510994    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhz8p\" (UniqueName: \"kubernetes.io/projected/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-kube-api-access-qhz8p\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.512889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.511031    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tls-certs\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.512889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.511080    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-model-cache\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.512889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.511117    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-dshm\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.512889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.511151    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tmp-dir\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.512889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.511657    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tmp-dir\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.512889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.512179    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-home\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.514050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.513766    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-model-cache\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.524573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.524537    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tls-certs\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.542591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.542520    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhz8p\" (UniqueName: \"kubernetes.io/projected/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-kube-api-access-qhz8p\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.558376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.556429    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-dshm\") pod \"simulator-w2-47dd19-kserve-744b66f76-sghdl\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") " pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.621400 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.621156    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:29.698880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.698859    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:25:29.699967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.699871    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:25:29.827371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.827104    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:30.003258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:29.999151    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"]
Nov 02 01:25:30.826915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:30.826880    4851 generic.go:356] "Generic (PLEG): container finished" podID="f8e45f01-7368-4448-bbec-2557dd42e004" containerID="2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5" exitCode=0
Nov 02 01:25:30.827546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:30.826957    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" event={"ID":"f8e45f01-7368-4448-bbec-2557dd42e004","Type":"ContainerDied","Data":"2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5"}
Nov 02 01:25:30.827546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:30.827446    4851 scope.go:122] "RemoveContainer" containerID="2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5"
Nov 02 01:25:30.828809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:30.828780    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:30.828906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:30.828887    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:30.828999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:30.828928    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:30.829723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:30.829703    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" event={"ID":"9fa8ee2f-b7c2-4394-8da0-b63261220c9b","Type":"ContainerStarted","Data":"f1f21637f77bc57322efb9d5b774991b26b35ac9202db396b7042ac74c715e21"}
Nov 02 01:25:30.829769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:30.829743    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" event={"ID":"9fa8ee2f-b7c2-4394-8da0-b63261220c9b","Type":"ContainerStarted","Data":"82c2385504e75a5c309f3957a0d80ad196c65eb84bbdc4b878d2f64b69f24b89"}
Nov 02 01:25:31.015318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.014622    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:31.193317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.183911    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:31.621731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.621701    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:31.716041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.716014    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:25:31.758313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.757730    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podStartSLOduration=2.757706769 podStartE2EDuration="2.757706769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:30.97290009 +0000 UTC m=+636.277832917" watchObservedRunningTime="2026-11-02 01:25:31.757706769 +0000 UTC m=+637.062639594"
Nov 02 01:25:31.885168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.870016    4851 generic.go:356] "Generic (PLEG): container finished" podID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerID="31b2bdd4e82af455230be053b83d6fbaed4cb1d93516a41beb780de730fcbeba" exitCode=0
Nov 02 01:25:31.885168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.870154    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" event={"ID":"a3849006-d06b-4c97-840d-5f34aa6cdfe1","Type":"ContainerDied","Data":"31b2bdd4e82af455230be053b83d6fbaed4cb1d93516a41beb780de730fcbeba"}
Nov 02 01:25:31.885168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.870704    4851 scope.go:122] "RemoveContainer" containerID="31b2bdd4e82af455230be053b83d6fbaed4cb1d93516a41beb780de730fcbeba"
Nov 02 01:25:31.897316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.889116    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:31.897316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.889212    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:31.897316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.889256    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:31.907219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:31.907184    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" event={"ID":"f8e45f01-7368-4448-bbec-2557dd42e004","Type":"ContainerStarted","Data":"28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5"}
Nov 02 01:25:32.087810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:32.087776    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:32.215362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:32.215320    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:32.865132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:32.865099    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:32.879821 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:32.879791    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:32.880178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:32.880091    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:25:32.880347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:32.880324    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 01:25:32.985312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:32.981999    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" event={"ID":"a3849006-d06b-4c97-840d-5f34aa6cdfe1","Type":"ContainerStarted","Data":"6ee873fff7889adb11a174b3521c5331556dfa29f04f1c0b3eecf344d9bce178"}
Nov 02 01:25:33.024209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:33.024173    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:33.200313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:33.196825    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:33.540944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:33.540804    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:33.762078 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:33.761846    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:33.883367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:33.877698    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:25:33.883367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:33.877755    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 01:25:33.905157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:33.904968    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:33.980330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:33.975134    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:33.980330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:33.975251    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:25:33.980330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:33.975314    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 02 01:25:34.139519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:34.137176    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:34.748028 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:34.747853    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:34.875167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:34.874716    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:25:34.875167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:34.874774    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 01:25:34.974854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:34.974402    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:25:34.974854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:34.974459    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 02 01:25:35.860316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:35.857166    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:35.871807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:35.871776    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:25:35.872093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:35.872053    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 01:25:35.983444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:35.983408    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:25:35.983973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:35.983944    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 02 01:25:36.882320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:36.875707    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:25:36.882320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:36.875769    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 01:25:36.976488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:36.976453    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:25:36.976757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:36.976729    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 02 01:25:37.075697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:37.075569    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:37.413028 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:37.412983    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:37.870974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:37.870884    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:25:37.870974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:37.870953    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Nov 02 01:25:37.973708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:37.973672    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:25:37.974160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:37.973739    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 02 01:25:38.165263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:38.165131    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:38.301892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:38.301832    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:38.439383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:38.438263    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:38.582145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:38.582102    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:38.877025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:38.876937    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:38.877356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:38.877339    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:38.878271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:38.878250    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:25:38.886130 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:38.886085    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:38.985543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:38.985512    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Nov 02 01:25:38.986060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:38.986032    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Nov 02 01:25:39.066176 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:39.066133    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:39.199584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:39.199550    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:39.329281 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:39.329239    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:39.622093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:39.622055    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:39.634773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:39.634746    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:39.702542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:39.702472    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:39.975098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:39.975071    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:39.975498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:39.975478    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:39.976072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:39.976056    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:25:40.045905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.045381    4851 generic.go:356] "Generic (PLEG): container finished" podID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerID="940927f8e46c5f52d34abf6b77bcd7d1e975c6dbf3c193eb2a358f3507425244" exitCode=0
Nov 02 01:25:40.045905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.045442    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" event={"ID":"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed","Type":"ContainerDied","Data":"940927f8e46c5f52d34abf6b77bcd7d1e975c6dbf3c193eb2a358f3507425244"}
Nov 02 01:25:40.046479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.045958    4851 scope.go:122] "RemoveContainer" containerID="940927f8e46c5f52d34abf6b77bcd7d1e975c6dbf3c193eb2a358f3507425244"
Nov 02 01:25:40.047633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.047606    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:40.047736 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.047686    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:40.047736 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.047723    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:40.050553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.050377    4851 generic.go:356] "Generic (PLEG): container finished" podID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerID="b68d48cf98a0c88d1052781961474a3cda98e2a273a98fce62672a7a0f898d1a" exitCode=0
Nov 02 01:25:40.051751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.050693    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" event={"ID":"93c39b84-1325-4b9f-8cbf-ef64ecf31237","Type":"ContainerDied","Data":"b68d48cf98a0c88d1052781961474a3cda98e2a273a98fce62672a7a0f898d1a"}
Nov 02 01:25:40.051751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.051491    4851 scope.go:122] "RemoveContainer" containerID="b68d48cf98a0c88d1052781961474a3cda98e2a273a98fce62672a7a0f898d1a"
Nov 02 01:25:40.057825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.057791    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:40.057958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.057876    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:40.057958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.057924    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:25:40.063388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.063364    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:25:40.814313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:40.814256    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:41.069901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.069803    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" event={"ID":"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed","Type":"ContainerStarted","Data":"32961c015c7e1cd2752244598c3af50ecd803b0fc75677cc9be8fce47746093a"}
Nov 02 01:25:41.079070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.079033    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" event={"ID":"93c39b84-1325-4b9f-8cbf-ef64ecf31237","Type":"ContainerStarted","Data":"c74e7d9b8d45a132b09303c041219fc71d7b8e8126c4a301f48e98a0259896d7"}
Nov 02 01:25:41.178316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.177982    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:41.180533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.180498    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:25:41.180797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.180771    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 02 01:25:41.338312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.338186    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 02 01:25:41.338598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.338555    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 02 01:25:41.338791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.338354    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:41.541132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.541076    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:41.584459 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.584426    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:41.584851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:41.584817    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:42.162358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.161892    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:25:42.162358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.161950    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 02 01:25:42.330044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.329998    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:42.336763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.336725    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 02 01:25:42.337050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.337024    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 02 01:25:42.516151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.516062    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"]
Nov 02 01:25:42.518905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.518883    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.522961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.522929    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ca1c53\"/\"kube-root-ca.crt\""
Nov 02 01:25:42.526211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.526187    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ca1c53\"/\"openshift-service-ca.crt\""
Nov 02 01:25:42.529971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.529945    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ca1c53\"/\"default-dockercfg-s6h7m\""
Nov 02 01:25:42.530095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.530074    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ca1c53\"/\"simulator-w0-ca1c53-kserve-self-signed-certs\""
Nov 02 01:25:42.562914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.562884    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"]
Nov 02 01:25:42.591760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.591728    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:42.695406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.694887    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-home\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.695406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.695004    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tmp-dir\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.695406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.695051    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz2zq\" (UniqueName: \"kubernetes.io/projected/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-kube-api-access-fz2zq\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.695406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.695080    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-dshm\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.695406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.695186    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-model-cache\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.695406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.695255    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tls-certs\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.797516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.796839    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-model-cache\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.797516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.796898    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tls-certs\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.797516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.796927    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-home\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.797516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.796972    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tmp-dir\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.797516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.796987    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2zq\" (UniqueName: \"kubernetes.io/projected/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-kube-api-access-fz2zq\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.797516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.797005    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-dshm\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.798221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.798196    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-home\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.799490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.798670    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-model-cache\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.799490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.799349    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tmp-dir\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.801020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.800995    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-dshm\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.801719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.801699    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tls-certs\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.813391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.813336    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2zq\" (UniqueName: \"kubernetes.io/projected/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-kube-api-access-fz2zq\") pod \"simulator-w0-ca1c53-kserve-55cdfdd467-ndqml\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") " pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:42.855340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:42.854505    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:43.185110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.184117    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:25:43.185110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.184183    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 02 01:25:43.185110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.185023    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:43.222749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.222724    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"]
Nov 02 01:25:43.225457 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:43.225415    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee158de2_d2d6_46dd_b0e8_b4e1a366633e.slice/crio-9d19de129c7fe907836c8230a646bfd5a6ec9eb741dc0f2cfb79a58095cddf9c WatchSource:0}: Error finding container 9d19de129c7fe907836c8230a646bfd5a6ec9eb741dc0f2cfb79a58095cddf9c: Status 404 returned error can't find the container with id 9d19de129c7fe907836c8230a646bfd5a6ec9eb741dc0f2cfb79a58095cddf9c
Nov 02 01:25:43.336267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.335809    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 02 01:25:43.336267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.335869    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 02 01:25:43.527909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.527878    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"]
Nov 02 01:25:43.530634 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.530603    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.570316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.570192    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5ed42b\"/\"kube-root-ca.crt\""
Nov 02 01:25:43.594323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.591624    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5ed42b\"/\"simulator-w3-5ed42b-kserve-self-signed-certs\""
Nov 02 01:25:43.594323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.592506    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"]
Nov 02 01:25:43.594323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.593127    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5ed42b\"/\"openshift-service-ca.crt\""
Nov 02 01:25:43.594323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.593201    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5ed42b\"/\"default-dockercfg-sflcv\""
Nov 02 01:25:43.627170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.627110    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:43.728317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.728208    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-model-cache\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.728317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.728282    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0126455-bcc3-4334-a73c-e53b3db5bce9-tls-certs\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.728559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.728336    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-dshm\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.728559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.728372    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-home\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.728559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.728429    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rcq\" (UniqueName: \"kubernetes.io/projected/a0126455-bcc3-4334-a73c-e53b3db5bce9-kube-api-access-s7rcq\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.728559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.728476    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-tmp-dir\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.886602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.846765    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0126455-bcc3-4334-a73c-e53b3db5bce9-tls-certs\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.886602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.846831    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-dshm\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.886602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.846880    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-home\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.886602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.846957    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rcq\" (UniqueName: \"kubernetes.io/projected/a0126455-bcc3-4334-a73c-e53b3db5bce9-kube-api-access-s7rcq\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.886602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.847026    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-tmp-dir\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.886602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.847118    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-model-cache\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.886602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.847627    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-model-cache\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.886602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.848610    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-home\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.886602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.849238    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-tmp-dir\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.886602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.881214    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0126455-bcc3-4334-a73c-e53b3db5bce9-tls-certs\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.942713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.933604    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-dshm\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:43.942713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:43.942060    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rcq\" (UniqueName: \"kubernetes.io/projected/a0126455-bcc3-4334-a73c-e53b3db5bce9-kube-api-access-s7rcq\") pod \"simulator-w3-5ed42b-kserve-656db64dbd-qn5fv\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") " pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:44.029316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.009758    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:44.126984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.126945    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" event={"ID":"ee158de2-d2d6-46dd-b0e8-b4e1a366633e","Type":"ContainerStarted","Data":"7a9b3efdd6bafb9fbbe8ddeea06ee828d5562e477105fdcb057d4806f2fd5fee"}
Nov 02 01:25:44.127231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.127216    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" event={"ID":"ee158de2-d2d6-46dd-b0e8-b4e1a366633e","Type":"ContainerStarted","Data":"9d19de129c7fe907836c8230a646bfd5a6ec9eb741dc0f2cfb79a58095cddf9c"}
Nov 02 01:25:44.166854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.166808    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:25:44.168497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.168462    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 02 01:25:44.225479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.224733    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:44.357772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.357732    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 02 01:25:44.358062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.358034    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 02 01:25:44.708322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.708228    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podStartSLOduration=2.708207105 podStartE2EDuration="2.708207105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:44.158239022 +0000 UTC m=+649.463171847" watchObservedRunningTime="2026-11-02 01:25:44.708207105 +0000 UTC m=+650.013139932"
Nov 02 01:25:44.709946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.709924    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"]
Nov 02 01:25:44.726993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.726963    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:44.875353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:44.864144    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:45.167844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:45.161699    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:25:45.167844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:45.161752    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 02 01:25:45.178142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:45.178042    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" event={"ID":"a0126455-bcc3-4334-a73c-e53b3db5bce9","Type":"ContainerStarted","Data":"cc4a2c0c9fe2d30255ea7bd57d6186a63b55c9ddd70f9c6d8de895f7a3e85510"}
Nov 02 01:25:45.178142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:45.178091    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" event={"ID":"a0126455-bcc3-4334-a73c-e53b3db5bce9","Type":"ContainerStarted","Data":"4d2188b1844f80635ef65eeabd7ec7fbe847818b81f4a24be000c7b9cb478fb6"}
Nov 02 01:25:45.198923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:45.198871    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podStartSLOduration=2.198855269 podStartE2EDuration="2.198855269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:45.197410353 +0000 UTC m=+650.502343180" watchObservedRunningTime="2026-11-02 01:25:45.198855269 +0000 UTC m=+650.503788095"
Nov 02 01:25:45.226421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:45.226385    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:45.348317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:45.341400    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 02 01:25:45.348317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:45.341463    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 02 01:25:45.447376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:45.445727    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:45.670315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:45.669815    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:46.161634 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:46.161601    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Nov 02 01:25:46.161827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:46.161682    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Nov 02 01:25:46.224840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:46.224807    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:46.419310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:46.419195    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:46.420904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:46.420881    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:25:46.826017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:46.825944    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:47.002269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:47.002236    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:47.165101 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:47.165073    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:47.166133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:47.166116    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:25:47.657970 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:25:47.657873    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:25:47.850247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:47.850209    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:48.410349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.410312    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"]
Nov 02 01:25:48.427237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.427209    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.433233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.433210    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b6ec07\"/\"openshift-service-ca.crt\""
Nov 02 01:25:48.436302 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.435525    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b6ec07\"/\"default-dockercfg-s95bt\""
Nov 02 01:25:48.436302 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.435584    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b6ec07\"/\"kube-root-ca.crt\""
Nov 02 01:25:48.436302 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.435750    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b6ec07\"/\"simulator-w4-b6ec07-kserve-self-signed-certs\""
Nov 02 01:25:48.437169 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.437142    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"]
Nov 02 01:25:48.455633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.455597    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8k2s\" (UniqueName: \"kubernetes.io/projected/94655091-3dc8-4b39-bcf2-a8fe7073b350-kube-api-access-q8k2s\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.455894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.455874    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94655091-3dc8-4b39-bcf2-a8fe7073b350-tls-certs\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.455985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.455949    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-dshm\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.456085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.456060    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-tmp-dir\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.456136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.456112    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-model-cache\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.456249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.456228    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-home\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.484502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.484472    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:48.557007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.556968    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94655091-3dc8-4b39-bcf2-a8fe7073b350-tls-certs\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.557216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.557058    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-dshm\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.557216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.557091    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-tmp-dir\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.557216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.557127    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-model-cache\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.557426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.557236    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-home\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.557426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.557282    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8k2s\" (UniqueName: \"kubernetes.io/projected/94655091-3dc8-4b39-bcf2-a8fe7073b350-kube-api-access-q8k2s\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.557933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.557881    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-model-cache\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.557933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.557923    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-tmp-dir\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.558118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.558044    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-home\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.559859 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.559839    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94655091-3dc8-4b39-bcf2-a8fe7073b350-tls-certs\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.560143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.560124    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-dshm\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.575413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.575386    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8k2s\" (UniqueName: \"kubernetes.io/projected/94655091-3dc8-4b39-bcf2-a8fe7073b350-kube-api-access-q8k2s\") pod \"simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") " pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:48.628235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.628199    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:48.746885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:48.746752    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:49.050647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:49.050615    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"]
Nov 02 01:25:49.050968 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:25:49.050936    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94655091_3dc8_4b39_bcf2_a8fe7073b350.slice/crio-ebb781aa640fd1ba77a8c3efa56339da67fe4106c5311543edf59035f4b9fdc3 WatchSource:0}: Error finding container ebb781aa640fd1ba77a8c3efa56339da67fe4106c5311543edf59035f4b9fdc3: Status 404 returned error can't find the container with id ebb781aa640fd1ba77a8c3efa56339da67fe4106c5311543edf59035f4b9fdc3
Nov 02 01:25:49.202065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:49.202030    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" event={"ID":"94655091-3dc8-4b39-bcf2-a8fe7073b350","Type":"ContainerStarted","Data":"5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2"}
Nov 02 01:25:49.202261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:49.202072    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" event={"ID":"94655091-3dc8-4b39-bcf2-a8fe7073b350","Type":"ContainerStarted","Data":"ebb781aa640fd1ba77a8c3efa56339da67fe4106c5311543edf59035f4b9fdc3"}
Nov 02 01:25:49.222077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:49.222027    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podStartSLOduration=1.22201153 podStartE2EDuration="1.22201153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:49.220905245 +0000 UTC m=+654.525838071" watchObservedRunningTime="2026-11-02 01:25:49.22201153 +0000 UTC m=+654.526944355"
Nov 02 01:25:49.336237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:49.336145    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:49.747936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:49.747900    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:50.033496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:50.033411    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:50.793621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:50.793586    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:51.522166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:51.522129    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:52.213847 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:52.213813    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:52.856562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:52.856531    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:52.869567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:52.869541    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:52.911227 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:52.911195    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:53.234016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:53.233984    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:25:53.636624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:53.636542    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:53.870226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:53.870192    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:54.226586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:54.226551    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:54.242946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:54.242915    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:54.329581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:54.329527    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:55.230873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:55.230815    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:55.330358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:55.330274    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:25:55.908200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:55.908165    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:56.599201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:56.599166    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:57.175745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:57.175706    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:57.534051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:57.533958    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:58.242111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:58.242074    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:58.747283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:58.747245    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:58.760157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:58.760128    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:58.942283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:58.942244    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:25:59.322763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:59.322735    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:25:59.632638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:25:59.632551    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:00.321078 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:00.321046    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:01.186856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.186811    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"]
Nov 02 01:26:01.191097 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.191067    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.191097 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.191076    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:01.201393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.201368    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"]
Nov 02 01:26:01.208093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.206534    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"kube-root-ca.crt\""
Nov 02 01:26:01.208093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.206668    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"openshift-service-ca.crt\""
Nov 02 01:26:01.208093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.207387    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"premium-w6-b9d816-kserve-self-signed-certs\""
Nov 02 01:26:01.208093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.207408    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"default-dockercfg-7z7ft\""
Nov 02 01:26:01.326853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.326806    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:01.393863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.393831    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-dshm\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.394067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.393931    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tmp-dir\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.394067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.393998    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5rgp\" (UniqueName: \"kubernetes.io/projected/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-kube-api-access-v5rgp\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.394067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.394049    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-model-cache\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.394246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.394091    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tls-certs\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.394246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.394152    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-home\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.494834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.494756    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-home\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.494834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.494829    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-dshm\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.495147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.494865    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tmp-dir\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.495147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.494911    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5rgp\" (UniqueName: \"kubernetes.io/projected/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-kube-api-access-v5rgp\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.495147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.494952    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-model-cache\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.495147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.494991    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tls-certs\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.495734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.495240    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-home\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.495734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.495398    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-model-cache\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.495734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.495600    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tmp-dir\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.497500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.497456    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-dshm\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.497998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.497978    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tls-certs\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.506977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.506955    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5rgp\" (UniqueName: \"kubernetes.io/projected/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-kube-api-access-v5rgp\") pod \"premium-w6-b9d816-kserve-765b7b6db-bppph\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") " pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.511954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.511927    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:01.796567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:01.796537    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"]
Nov 02 01:26:01.798968 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:26:01.798941    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7b8ea2a_5325_4ed7_8988_2c46ec6eda1d.slice/crio-2eb38df7f3935a6c3c2eba274dc8e494f082df91f7557a7ba1e9d5070b960e04 WatchSource:0}: Error finding container 2eb38df7f3935a6c3c2eba274dc8e494f082df91f7557a7ba1e9d5070b960e04: Status 404 returned error can't find the container with id 2eb38df7f3935a6c3c2eba274dc8e494f082df91f7557a7ba1e9d5070b960e04
Nov 02 01:26:02.143989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:02.143901    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:02.330580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:02.330538    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" event={"ID":"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d","Type":"ContainerStarted","Data":"221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7"}
Nov 02 01:26:02.330580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:02.330592    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" event={"ID":"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d","Type":"ContainerStarted","Data":"2eb38df7f3935a6c3c2eba274dc8e494f082df91f7557a7ba1e9d5070b960e04"}
Nov 02 01:26:02.350573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:02.350525    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podStartSLOduration=1.350511139 podStartE2EDuration="1.350511139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:02.348861678 +0000 UTC m=+667.653794505" watchObservedRunningTime="2026-11-02 01:26:02.350511139 +0000 UTC m=+667.655443964"
Nov 02 01:26:02.512213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:02.512176    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:02.925537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:02.925506    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:03.806365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:03.806329    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:04.608202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:04.608162    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:05.419566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.419497    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:05.524255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.524227    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"]
Nov 02 01:26:05.526992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.526966    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.531608 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.531586    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ca1c53\"/\"premium-w0-ca1c53-kserve-self-signed-certs\""
Nov 02 01:26:05.538065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.538040    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"]
Nov 02 01:26:05.592097 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.592063    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:05.676328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.670560    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-tmp-dir\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.676328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.670672    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-dshm\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.676328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.670720    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps7vl\" (UniqueName: \"kubernetes.io/projected/17113a57-21c8-46c1-a80a-deb91dc51a87-kube-api-access-ps7vl\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.676328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.670754    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-home\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.676328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.670779    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17113a57-21c8-46c1-a80a-deb91dc51a87-tls-certs\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.676328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.670803    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-model-cache\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.772196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.772156    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-tmp-dir\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.772434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.772264    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-dshm\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.772434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.772342    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps7vl\" (UniqueName: \"kubernetes.io/projected/17113a57-21c8-46c1-a80a-deb91dc51a87-kube-api-access-ps7vl\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.772434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.772372    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-home\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.772434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.772397    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17113a57-21c8-46c1-a80a-deb91dc51a87-tls-certs\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.772434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.772419    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-model-cache\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.772722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.772697    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-tmp-dir\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.772775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.772728    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-home\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.772775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.772768    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-model-cache\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.774751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.774730    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-dshm\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.775888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.775866    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17113a57-21c8-46c1-a80a-deb91dc51a87-tls-certs\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.785572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.785544    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps7vl\" (UniqueName: \"kubernetes.io/projected/17113a57-21c8-46c1-a80a-deb91dc51a87-kube-api-access-ps7vl\") pod \"premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") " pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:05.838912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:05.838866    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:06.116834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:06.116798    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"]
Nov 02 01:26:06.118223 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:26:06.118199    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17113a57_21c8_46c1_a80a_deb91dc51a87.slice/crio-e410af5f5cf142c96ac01438a7c8fd1fc1cae91a0e99fe8b8294a75106847a2f WatchSource:0}: Error finding container e410af5f5cf142c96ac01438a7c8fd1fc1cae91a0e99fe8b8294a75106847a2f: Status 404 returned error can't find the container with id e410af5f5cf142c96ac01438a7c8fd1fc1cae91a0e99fe8b8294a75106847a2f
Nov 02 01:26:06.220921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:06.220886    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:06.354003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:06.353965    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" event={"ID":"17113a57-21c8-46c1-a80a-deb91dc51a87","Type":"ContainerStarted","Data":"fbb13aff88a39fc3da0c8b159d30baefcfe011caba85f9f8bdbd93f57fef556d"}
Nov 02 01:26:06.354003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:06.354006    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" event={"ID":"17113a57-21c8-46c1-a80a-deb91dc51a87","Type":"ContainerStarted","Data":"e410af5f5cf142c96ac01438a7c8fd1fc1cae91a0e99fe8b8294a75106847a2f"}
Nov 02 01:26:06.376810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:06.376711    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podStartSLOduration=1.376696757 podStartE2EDuration="1.376696757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:06.375027351 +0000 UTC m=+671.679960190" watchObservedRunningTime="2026-11-02 01:26:06.376696757 +0000 UTC m=+671.681629582"
Nov 02 01:26:06.839262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:06.839223    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:07.016031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:07.015991    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:08.022283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:08.022243    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:08.727647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:08.727612    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:09.441408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:09.441364    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:09.868357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:09.868233    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:10.369811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:10.369780    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:11.222791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:11.222747    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:11.512769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:11.512684    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:11.525274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:11.525247    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:11.924708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:11.924678    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:12.401357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:12.401261    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:26:12.670441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:12.670407    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:12.797912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:12.797866    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:13.545717 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:13.545681    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:14.404074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:14.404038    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:15.170504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.170467    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:15.401898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.401863    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"]
Nov 02 01:26:15.405100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.405076    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.411994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.411964    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b6ec07\"/\"premium-w4-b6ec07-kserve-self-signed-certs\""
Nov 02 01:26:15.417713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.417683    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"]
Nov 02 01:26:15.481474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.481394    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:15.519572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.519532    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqcps\" (UniqueName: \"kubernetes.io/projected/1afe6338-3fb4-4829-88f9-232fc8c32127-kube-api-access-gqcps\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.519768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.519671    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-tmp-dir\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.519768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.519729    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-model-cache\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.519900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.519767    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-dshm\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.519900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.519869    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1afe6338-3fb4-4829-88f9-232fc8c32127-tls-certs\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.519999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.519971    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-home\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.621355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.621316    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-tmp-dir\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.621561 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.621383    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-model-cache\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.621561 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.621424    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-dshm\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.621561 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.621453    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1afe6338-3fb4-4829-88f9-232fc8c32127-tls-certs\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.621561 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.621508    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-home\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.621773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.621593    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqcps\" (UniqueName: \"kubernetes.io/projected/1afe6338-3fb4-4829-88f9-232fc8c32127-kube-api-access-gqcps\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.621773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.621732    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-model-cache\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.621877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.621816    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-tmp-dir\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.621926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.621905    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-home\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.623891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.623869    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-dshm\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.624222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.624205    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1afe6338-3fb4-4829-88f9-232fc8c32127-tls-certs\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.633582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.633562    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqcps\" (UniqueName: \"kubernetes.io/projected/1afe6338-3fb4-4829-88f9-232fc8c32127-kube-api-access-gqcps\") pod \"premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") " pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.720038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.720002    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:15.839940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.839904    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:15.853355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.853328    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:15.929701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:15.929672    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:16.008161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:16.008135    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"]
Nov 02 01:26:16.009934 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:26:16.009907    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1afe6338_3fb4_4829_88f9_232fc8c32127.slice/crio-0049eb4585bb3222e4e0c7b5344e791c49e9735ad53943be62087bc6d28eae07 WatchSource:0}: Error finding container 0049eb4585bb3222e4e0c7b5344e791c49e9735ad53943be62087bc6d28eae07: Status 404 returned error can't find the container with id 0049eb4585bb3222e4e0c7b5344e791c49e9735ad53943be62087bc6d28eae07
Nov 02 01:26:16.419114 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:16.419084    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" event={"ID":"1afe6338-3fb4-4829-88f9-232fc8c32127","Type":"ContainerStarted","Data":"46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4"}
Nov 02 01:26:16.419630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:16.419128    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" event={"ID":"1afe6338-3fb4-4829-88f9-232fc8c32127","Type":"ContainerStarted","Data":"0049eb4585bb3222e4e0c7b5344e791c49e9735ad53943be62087bc6d28eae07"}
Nov 02 01:26:16.430699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:16.430678    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:26:16.438596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:16.438543    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podStartSLOduration=1.438531068 podStartE2EDuration="1.438531068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:16.437240768 +0000 UTC m=+681.742173593" watchObservedRunningTime="2026-11-02 01:26:16.438531068 +0000 UTC m=+681.743463893"
Nov 02 01:26:16.707721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:16.707641    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:16.727625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:16.723574    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:17.416639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:17.416597    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:17.654906 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:26:17.654871    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:26:18.131955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:18.131926    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:18.816371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:18.816335    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:19.529704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:19.529670    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:20.219980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:20.219952    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:20.926200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:20.926167    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:21.838911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:21.838873    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:22.893018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:22.892960    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:23.875881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:23.875846    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"]
Nov 02 01:26:23.886237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:23.886208    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:23.892116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:23.892082    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"]
Nov 02 01:26:23.892262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:23.892155    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-47dd19-openshift-default-dockercfg-tx9jn\""
Nov 02 01:26:24.031173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.031137    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:24.041773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.041726    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mcz8\" (UniqueName: \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-kube-api-access-5mcz8\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.042186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.042160    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.042396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.042378    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d13070b-3087-452c-a330-c5179e3bd0cf-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.042543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.042528    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.043123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.043099    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.048306 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.043347    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.048598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.048574    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.048818 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.048792    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.048974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.048857    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.149749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.149663    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.149749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.149715    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.149975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.149757    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mcz8\" (UniqueName: \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-kube-api-access-5mcz8\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.149975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.149804    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.149975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.149832    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d13070b-3087-452c-a330-c5179e3bd0cf-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.149975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.149870    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.149975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.149926    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.150254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.149989    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.150254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.150051    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.150368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.150206    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.150719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.150687    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d13070b-3087-452c-a330-c5179e3bd0cf-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.151154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.151129    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.151407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.151197    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.151647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.151626    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.153341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.153318    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.153535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.153514    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.162363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.162344    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.162752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.162728    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mcz8\" (UniqueName: \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-kube-api-access-5mcz8\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.214826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.214793    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:24.507601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.507569    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"]
Nov 02 01:26:24.508068 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:26:24.508033    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d13070b_3087_452c_a330_c5179e3bd0cf.slice/crio-62aa3e49af6ba33fd005289603625f8d94778eeb7741855864a660dc69af6110 WatchSource:0}: Error finding container 62aa3e49af6ba33fd005289603625f8d94778eeb7741855864a660dc69af6110: Status 404 returned error can't find the container with id 62aa3e49af6ba33fd005289603625f8d94778eeb7741855864a660dc69af6110
Nov 02 01:26:24.510746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.510704    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:26:24.510881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.510795    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:26:24.510881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.510838    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:26:24.729933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:24.729903    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:25.522707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:25.522669    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" event={"ID":"8d13070b-3087-452c-a330-c5179e3bd0cf","Type":"ContainerStarted","Data":"0883378e5dd8828c38bf94c41b1a2dce7ae12490afbc2d90417f9574b50f2692"}
Nov 02 01:26:25.523193 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:25.522721    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" event={"ID":"8d13070b-3087-452c-a330-c5179e3bd0cf","Type":"ContainerStarted","Data":"62aa3e49af6ba33fd005289603625f8d94778eeb7741855864a660dc69af6110"}
Nov 02 01:26:25.523653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:25.523623    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:25.572317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:25.567948    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" podStartSLOduration=2.567928031 podStartE2EDuration="2.567928031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:25.557851927 +0000 UTC m=+690.862784754" watchObservedRunningTime="2026-11-02 01:26:25.567928031 +0000 UTC m=+690.872860858"
Nov 02 01:26:25.721203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:25.721165    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:25.758701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:25.758661    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:26.215282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:26.215240    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:26.217055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:26.217027    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:26:26.217193 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:26.217076    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" podUID="8d13070b-3087-452c-a330-c5179e3bd0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:26:26.257325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:26.256025    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:26.566461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:26.566375    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:26:26.961305 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:26.960753    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:27.215222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:27.215127    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:26:27.215222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:27.215187    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" podUID="8d13070b-3087-452c-a330-c5179e3bd0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:26:27.914681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:27.914642    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:28.215907 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:28.215825    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:26:28.215907 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:28.215878    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" podUID="8d13070b-3087-452c-a330-c5179e3bd0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:26:28.914579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:28.914546    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:29.215696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:29.215605    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:26:29.215696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:29.215663    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" podUID="8d13070b-3087-452c-a330-c5179e3bd0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:26:29.712012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:29.711971    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:30.215401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:30.215371    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 01:26:30.215581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:30.215430    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" podUID="8d13070b-3087-452c-a330-c5179e3bd0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 01:26:30.394711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:30.394575    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:31.044501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:31.044460    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:31.446450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:31.446419    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:31.447261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:31.447241    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:31.448509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:31.448491    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:26:31.746352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:31.746247    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:32.482333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:32.481283    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:33.181531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:33.181502    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:33.918700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:33.918671    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:34.628042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:34.628011    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:35.508716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:35.508680    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:36.433118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:36.433076    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:37.324316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:37.321451    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:38.127651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:38.127619    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:39.182677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.182641    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"]
Nov 02 01:26:39.186384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.186367    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.193439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.193414    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-123986-openshift-default-dockercfg-8gxrz\""
Nov 02 01:26:39.203498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.203469    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"]
Nov 02 01:26:39.216662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.216637    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:39.247914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.247885    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"]
Nov 02 01:26:39.251708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.251689    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.265856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.265825    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"]
Nov 02 01:26:39.291912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.291882    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ca1c53-openshift-default-dockercfg-hhbph\""
Nov 02 01:26:39.310413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310375    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.310413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310419    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.310677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310437    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcqdq\" (UniqueName: \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-kube-api-access-jcqdq\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.310677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310534    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fphq9\" (UniqueName: \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-kube-api-access-fphq9\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.310677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310582    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.310677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310599    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.310677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310634    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a26fed-b9e3-4085-9f05-d33b5532a9df-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.310969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310689    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.310969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310711    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.310969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310753    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.310969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310769    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.310969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310784    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.310969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310805    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.310969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310821    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.310969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310887    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.310969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.310948    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1aa882-4253-4421-a52c-d300b159fb5c-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.311480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.311045    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.311480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.311078    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.412536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412502    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.412536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412538    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.412787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412576    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.412787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412607    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.412787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412623    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcqdq\" (UniqueName: \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-kube-api-access-jcqdq\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.412787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412648    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fphq9\" (UniqueName: \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-kube-api-access-fphq9\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.412787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412688    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.412787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412713    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.412787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412767    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a26fed-b9e3-4085-9f05-d33b5532a9df-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412809    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412839    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412877    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412901    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412925    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412961    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.412993    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413003    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413029    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413060    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1aa882-4253-4421-a52c-d300b159fb5c-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413064    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413102    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.413743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413159    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.413743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413285    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.413743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413356    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.413743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413548    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.414121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413816    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.414121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.413844    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a26fed-b9e3-4085-9f05-d33b5532a9df-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.414234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.414132    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1aa882-4253-4421-a52c-d300b159fb5c-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.415663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.415639    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.415858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.415842    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.416161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.416140    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.416465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.416444    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.425498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.425470    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fphq9\" (UniqueName: \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-kube-api-access-fphq9\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.426107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.426062    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcqdq\" (UniqueName: \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-kube-api-access-jcqdq\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.426229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.426186    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.426595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.426575    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.503704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.503623    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:39.570174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.569654    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:39.796550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.796518    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"]
Nov 02 01:26:39.796695 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:26:39.796671    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1aa882_4253_4421_a52c_d300b159fb5c.slice/crio-382f44a8800f8523a1d255b7ff4514887cb9b5278b091d6b402cf54693a962dd WatchSource:0}: Error finding container 382f44a8800f8523a1d255b7ff4514887cb9b5278b091d6b402cf54693a962dd: Status 404 returned error can't find the container with id 382f44a8800f8523a1d255b7ff4514887cb9b5278b091d6b402cf54693a962dd
Nov 02 01:26:39.799352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.799322    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:26:39.799468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.799403    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:26:39.799468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.799449    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:26:39.882944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.882915    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"]
Nov 02 01:26:39.883138 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:26:39.883105    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a26fed_b9e3_4085_9f05_d33b5532a9df.slice/crio-6e97d1ee287654e729798c2eafc72341d8f6ee54b4f602f75b533dd3546e0b19 WatchSource:0}: Error finding container 6e97d1ee287654e729798c2eafc72341d8f6ee54b4f602f75b533dd3546e0b19: Status 404 returned error can't find the container with id 6e97d1ee287654e729798c2eafc72341d8f6ee54b4f602f75b533dd3546e0b19
Nov 02 01:26:39.886450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.886418    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:26:39.886564 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.886493    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:26:39.886564 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:39.886526    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:26:40.369314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:40.368649    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:40.693312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:40.693195    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" event={"ID":"1d1aa882-4253-4421-a52c-d300b159fb5c","Type":"ContainerStarted","Data":"6ae2f75c860e68b4d05eb578a0e5644cc6721851df195f481db2f1e3aa728480"}
Nov 02 01:26:40.693312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:40.693265    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" event={"ID":"1d1aa882-4253-4421-a52c-d300b159fb5c","Type":"ContainerStarted","Data":"382f44a8800f8523a1d255b7ff4514887cb9b5278b091d6b402cf54693a962dd"}
Nov 02 01:26:40.697312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:40.696527    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" event={"ID":"39a26fed-b9e3-4085-9f05-d33b5532a9df","Type":"ContainerStarted","Data":"da16952a12914a9d6a8f09bbac314a86b7a1d5e837c1b8e64a93803e7c492ee1"}
Nov 02 01:26:40.697312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:40.696586    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" event={"ID":"39a26fed-b9e3-4085-9f05-d33b5532a9df","Type":"ContainerStarted","Data":"6e97d1ee287654e729798c2eafc72341d8f6ee54b4f602f75b533dd3546e0b19"}
Nov 02 01:26:40.730317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:40.728342    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" podStartSLOduration=1.728323719 podStartE2EDuration="1.728323719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:40.721735663 +0000 UTC m=+706.026668500" watchObservedRunningTime="2026-11-02 01:26:40.728323719 +0000 UTC m=+706.033256544"
Nov 02 01:26:41.287327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:41.286953    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:41.505003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:41.504401    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:41.506796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:41.506761    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:26:41.506927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:41.506810    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 01:26:41.573387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:41.572608    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:41.585338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:41.583410    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:26:41.585338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:41.583473    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" podUID="39a26fed-b9e3-4085-9f05-d33b5532a9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 01:26:42.258327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:42.254773    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:42.504805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:42.504765    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:26:42.505016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:42.504821    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 01:26:42.576377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:42.574826    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:26:42.576377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:42.574897    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" podUID="39a26fed-b9e3-4085-9f05-d33b5532a9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 01:26:43.066318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:43.065939    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:43.504803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:43.504766    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:26:43.504985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:43.504818    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 01:26:43.579315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:43.573707    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:26:43.579315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:43.573770    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" podUID="39a26fed-b9e3-4085-9f05-d33b5532a9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 01:26:43.986317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:43.985248    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:44.504905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:44.504863    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:26:44.505117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:44.504912    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 01:26:44.572315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:44.571726    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:26:44.572315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:44.571787    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" podUID="39a26fed-b9e3-4085-9f05-d33b5532a9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 01:26:44.929213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:44.929184    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:45.504311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:45.504240    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:26:45.504510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:45.504338    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 01:26:45.582330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:45.578413    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:26:45.582330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:45.578472    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" podUID="39a26fed-b9e3-4085-9f05-d33b5532a9df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 01:26:45.644317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:45.638212    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:45.960845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:45.960807    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:46.677050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:46.676906    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:46.734899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:46.734857    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:46.738417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:46.738393    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:46.768079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:46.761601    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:26:46.779233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:46.779171    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" podStartSLOduration=7.779152817 podStartE2EDuration="7.779152817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:40.757376592 +0000 UTC m=+706.062309422" watchObservedRunningTime="2026-11-02 01:26:46.779152817 +0000 UTC m=+712.084085645"
Nov 02 01:26:46.939160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:46.939024    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:46.940544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:46.940149    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:46.941121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:46.940956    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:26:47.594067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:47.594029    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:47.655543 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:26:47.655511    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:26:48.391496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:48.391457    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:49.107126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:49.107095    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:49.919683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:49.919652    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:50.527556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.527511    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"]
Nov 02 01:26:50.532878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.532859    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:50.539011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.538475    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:26:50.539011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.538530    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:26:50.539011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.538928    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:26:50.539439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.539420    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:26:50.541340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.541318    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"]
Nov 02 01:26:50.656775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.656742    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlh5m\" (UniqueName: \"kubernetes.io/projected/b547c344-04eb-4df1-80c1-575bf46d3f3d-kube-api-access-mlh5m\") pod \"maas-api-revoke-sub-4c6dc38c-lpnsh\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:50.656775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.656776    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6dc38c-lpnsh\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:50.656995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.656970    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6dc38c-lpnsh\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:50.713211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.713182    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:50.759985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.759944    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6dc38c-lpnsh\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:50.760320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.760280    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlh5m\" (UniqueName: \"kubernetes.io/projected/b547c344-04eb-4df1-80c1-575bf46d3f3d-kube-api-access-mlh5m\") pod \"maas-api-revoke-sub-4c6dc38c-lpnsh\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:50.760454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.760438    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6dc38c-lpnsh\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:50.761093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.761069    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6dc38c-lpnsh\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:50.776364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.772847    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6dc38c-lpnsh\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:50.783782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.783612    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlh5m\" (UniqueName: \"kubernetes.io/projected/b547c344-04eb-4df1-80c1-575bf46d3f3d-kube-api-access-mlh5m\") pod \"maas-api-revoke-sub-4c6dc38c-lpnsh\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:50.848810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:50.848783    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:51.124148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:51.124123    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"]
Nov 02 01:26:51.126015 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:26:51.125987    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb547c344_04eb_4df1_80c1_575bf46d3f3d.slice/crio-661012b37a2979b00008ac80c9adb772e49422d0c3021c967b0451d3478ad558 WatchSource:0}: Error finding container 661012b37a2979b00008ac80c9adb772e49422d0c3021c967b0451d3478ad558: Status 404 returned error can't find the container with id 661012b37a2979b00008ac80c9adb772e49422d0c3021c967b0451d3478ad558
Nov 02 01:26:51.127942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:51.127924    4851 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:26:51.416570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:51.416527    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:51.792584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:51.792493    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh" event={"ID":"b547c344-04eb-4df1-80c1-575bf46d3f3d","Type":"ContainerStarted","Data":"661012b37a2979b00008ac80c9adb772e49422d0c3021c967b0451d3478ad558"}
Nov 02 01:26:52.119376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:52.119272    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:52.812127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:52.812085    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:53.714317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:53.711882    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:54.414410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.409881    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"]
Nov 02 01:26:54.421120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.421091    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"]
Nov 02 01:26:54.421282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.421075    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.564316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.563261    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:54.592772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.592732    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4xj\" (UniqueName: \"kubernetes.io/projected/80d78919-d93b-4a18-8dca-f5034b63d756-kube-api-access-rb4xj\") pod \"maas-api-revoke-sub-49e0ad96-xpjhk\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.593068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.593046    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e0ad96-xpjhk\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.593531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.593500    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e0ad96-xpjhk\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.708669 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.698965    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e0ad96-xpjhk\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.708669 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.699071    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4xj\" (UniqueName: \"kubernetes.io/projected/80d78919-d93b-4a18-8dca-f5034b63d756-kube-api-access-rb4xj\") pod \"maas-api-revoke-sub-49e0ad96-xpjhk\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.708669 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.699132    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e0ad96-xpjhk\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.708669 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.700125    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e0ad96-xpjhk\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.709446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.709417    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e0ad96-xpjhk\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.715113 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.715069    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb4xj\" (UniqueName: \"kubernetes.io/projected/80d78919-d93b-4a18-8dca-f5034b63d756-kube-api-access-rb4xj\") pod \"maas-api-revoke-sub-49e0ad96-xpjhk\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.751864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.751838    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:54.990490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:54.990401    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"]
Nov 02 01:26:55.006568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.006537    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"]
Nov 02 01:26:55.006733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.006688    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.069331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.069128    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"]
Nov 02 01:26:55.071725 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:26:55.071693    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d78919_d93b_4a18_8dca_f5034b63d756.slice/crio-79892cd40d425ca92d58110e5c1e65b323bc5e7cb06cccb5ab41288e248b494d WatchSource:0}: Error finding container 79892cd40d425ca92d58110e5c1e65b323bc5e7cb06cccb5ab41288e248b494d: Status 404 returned error can't find the container with id 79892cd40d425ca92d58110e5c1e65b323bc5e7cb06cccb5ab41288e248b494d
Nov 02 01:26:55.106954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.106909    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59jfr\" (UniqueName: \"kubernetes.io/projected/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-kube-api-access-59jfr\") pod \"maas-api-revoke-sub-a1628473-m7q9j\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.106954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.106951    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1628473-m7q9j\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.107173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.107065    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1628473-m7q9j\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.208609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.208570    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1628473-m7q9j\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.208806 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.208756    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59jfr\" (UniqueName: \"kubernetes.io/projected/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-kube-api-access-59jfr\") pod \"maas-api-revoke-sub-a1628473-m7q9j\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.208806 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.208785    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1628473-m7q9j\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.209723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.209668    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1628473-m7q9j\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.211925 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.211905    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1628473-m7q9j\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.231574 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.231542    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59jfr\" (UniqueName: \"kubernetes.io/projected/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-kube-api-access-59jfr\") pod \"maas-api-revoke-sub-a1628473-m7q9j\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.332170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.331779    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:55.521105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.521069    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:55.629813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.629780    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"]
Nov 02 01:26:55.854095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.854056    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j" event={"ID":"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51","Type":"ContainerStarted","Data":"53d5a74ea3d04e2920f33c9cf34998f2a2e7484369ee1a1999d72d34c2597787"}
Nov 02 01:26:55.855766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:55.855726    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk" event={"ID":"80d78919-d93b-4a18-8dca-f5034b63d756","Type":"ContainerStarted","Data":"79892cd40d425ca92d58110e5c1e65b323bc5e7cb06cccb5ab41288e248b494d"}
Nov 02 01:26:56.338057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:56.338028    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:56.887275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:56.887221    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh" event={"ID":"b547c344-04eb-4df1-80c1-575bf46d3f3d","Type":"ContainerStarted","Data":"b714e9784bac376a2d975ac86f5898f87378effc2a6983259e67494be21dcd95"}
Nov 02 01:26:56.889826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:56.889792    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk" event={"ID":"80d78919-d93b-4a18-8dca-f5034b63d756","Type":"ContainerStarted","Data":"fc02b8ce3b478bab917ddd27179f6a56720bb6bdb29383fb2523eb5531bddecc"}
Nov 02 01:26:56.925601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:56.925552    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk" podStartSLOduration=1.6789266729999999 podStartE2EDuration="2.925538189s" totalImagesPullingTime="1.246611516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:56.924424735 +0000 UTC m=+722.229357563" watchObservedRunningTime="2026-11-02 01:26:56.925538189 +0000 UTC m=+722.230471015"
Nov 02 01:26:57.164860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:57.164809    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:57.904384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:57.904343    4851 generic.go:356] "Generic (PLEG): container finished" podID="df2e8e49-e371-4d51-88e8-3fe8e3b3ea51" containerID="af4cb02bad9a38ddce1ce1a4c966e07e4f674896c880e822fec51eacea6091b9" exitCode=0
Nov 02 01:26:57.904899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:57.904425    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j" event={"ID":"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51","Type":"ContainerDied","Data":"af4cb02bad9a38ddce1ce1a4c966e07e4f674896c880e822fec51eacea6091b9"}
Nov 02 01:26:57.906257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:57.906233    4851 generic.go:356] "Generic (PLEG): container finished" podID="b547c344-04eb-4df1-80c1-575bf46d3f3d" containerID="b714e9784bac376a2d975ac86f5898f87378effc2a6983259e67494be21dcd95" exitCode=0
Nov 02 01:26:57.906431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:57.906340    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh" event={"ID":"b547c344-04eb-4df1-80c1-575bf46d3f3d","Type":"ContainerDied","Data":"b714e9784bac376a2d975ac86f5898f87378effc2a6983259e67494be21dcd95"}
Nov 02 01:26:57.908161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:57.908141    4851 generic.go:356] "Generic (PLEG): container finished" podID="80d78919-d93b-4a18-8dca-f5034b63d756" containerID="fc02b8ce3b478bab917ddd27179f6a56720bb6bdb29383fb2523eb5531bddecc" exitCode=0
Nov 02 01:26:57.908280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:57.908202    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk" event={"ID":"80d78919-d93b-4a18-8dca-f5034b63d756","Type":"ContainerDied","Data":"fc02b8ce3b478bab917ddd27179f6a56720bb6bdb29383fb2523eb5531bddecc"}
Nov 02 01:26:58.067466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.067393    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:58.138300 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.138258    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:58.153773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.153743    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-openshift-ca\") pod \"b547c344-04eb-4df1-80c1-575bf46d3f3d\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") "
Nov 02 01:26:58.153940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.153825    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b547c344-04eb-4df1-80c1-575bf46d3f3d-kube-api-access-mlh5m\" (UniqueName: \"kubernetes.io/projected/b547c344-04eb-4df1-80c1-575bf46d3f3d-kube-api-access-mlh5m\") pod \"b547c344-04eb-4df1-80c1-575bf46d3f3d\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") "
Nov 02 01:26:58.154008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.153960    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-xks-ca\") pod \"b547c344-04eb-4df1-80c1-575bf46d3f3d\" (UID: \"b547c344-04eb-4df1-80c1-575bf46d3f3d\") "
Nov 02 01:26:58.154148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.154122    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-openshift-ca" pod "b547c344-04eb-4df1-80c1-575bf46d3f3d" (UID: "b547c344-04eb-4df1-80c1-575bf46d3f3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:58.154428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.154411    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:26:58.156249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.156220    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-xks-ca" pod "b547c344-04eb-4df1-80c1-575bf46d3f3d" (UID: "b547c344-04eb-4df1-80c1-575bf46d3f3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:58.156359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.156219    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b547c344-04eb-4df1-80c1-575bf46d3f3d-kube-api-access-mlh5m" pod "b547c344-04eb-4df1-80c1-575bf46d3f3d" (UID: "b547c344-04eb-4df1-80c1-575bf46d3f3d"). InnerVolumeSpecName "kube-api-access-mlh5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:26:58.255463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.255431    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b547c344-04eb-4df1-80c1-575bf46d3f3d-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:26:58.255463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.255457    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlh5m\" (UniqueName: \"kubernetes.io/projected/b547c344-04eb-4df1-80c1-575bf46d3f3d-kube-api-access-mlh5m\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:26:58.336952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.336874    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"]
Nov 02 01:26:58.337849 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.337817    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b547c344-04eb-4df1-80c1-575bf46d3f3d" containerName="revoke-keys"
Nov 02 01:26:58.337849 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.337845    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b547c344-04eb-4df1-80c1-575bf46d3f3d" containerName="revoke-keys"
Nov 02 01:26:58.338003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.337990    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b547c344-04eb-4df1-80c1-575bf46d3f3d" containerName="revoke-keys"
Nov 02 01:26:58.343630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.343602    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.359749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.359718    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"]
Nov 02 01:26:58.459545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.459505    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5ec0c2-z8rkv\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.459744 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.459662    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm27g\" (UniqueName: \"kubernetes.io/projected/a53319cf-e568-412f-be8c-cd5b056ff07e-kube-api-access-bm27g\") pod \"maas-api-revoke-sub-dd5ec0c2-z8rkv\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.459812 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.459767    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5ec0c2-z8rkv\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.560930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.560892    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5ec0c2-z8rkv\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.561129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.560941    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm27g\" (UniqueName: \"kubernetes.io/projected/a53319cf-e568-412f-be8c-cd5b056ff07e-kube-api-access-bm27g\") pod \"maas-api-revoke-sub-dd5ec0c2-z8rkv\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.561129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.560975    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5ec0c2-z8rkv\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.561766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.561741    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5ec0c2-z8rkv\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.563532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.563513    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5ec0c2-z8rkv\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.574463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.574444    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm27g\" (UniqueName: \"kubernetes.io/projected/a53319cf-e568-412f-be8c-cd5b056ff07e-kube-api-access-bm27g\") pod \"maas-api-revoke-sub-dd5ec0c2-z8rkv\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.687372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.676475    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:26:58.916141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.916109    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"
Nov 02 01:26:58.916569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.916146    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh" event={"ID":"b547c344-04eb-4df1-80c1-575bf46d3f3d","Type":"ContainerDied","Data":"661012b37a2979b00008ac80c9adb772e49422d0c3021c967b0451d3478ad558"}
Nov 02 01:26:58.916569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:58.916200    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="661012b37a2979b00008ac80c9adb772e49422d0c3021c967b0451d3478ad558"
Nov 02 01:26:59.036085 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:26:59.035407    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53319cf_e568_412f_be8c_cd5b056ff07e.slice/crio-f2b98f150654ffbcc4ac156cf930a3177030180e5f51599fbb887d2a94f89dbf WatchSource:0}: Error finding container f2b98f150654ffbcc4ac156cf930a3177030180e5f51599fbb887d2a94f89dbf: Status 404 returned error can't find the container with id f2b98f150654ffbcc4ac156cf930a3177030180e5f51599fbb887d2a94f89dbf
Nov 02 01:26:59.040310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.039942    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"]
Nov 02 01:26:59.060315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.057534    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:26:59.193315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.190999    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:26:59.312279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.312219    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-xks-ca\") pod \"80d78919-d93b-4a18-8dca-f5034b63d756\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") "
Nov 02 01:26:59.312463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.312337    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d78919-d93b-4a18-8dca-f5034b63d756-kube-api-access-rb4xj\" (UniqueName: \"kubernetes.io/projected/80d78919-d93b-4a18-8dca-f5034b63d756-kube-api-access-rb4xj\") pod \"80d78919-d93b-4a18-8dca-f5034b63d756\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") "
Nov 02 01:26:59.312611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.312594    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-openshift-ca\") pod \"80d78919-d93b-4a18-8dca-f5034b63d756\" (UID: \"80d78919-d93b-4a18-8dca-f5034b63d756\") "
Nov 02 01:26:59.317106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.316820    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-xks-ca" pod "80d78919-d93b-4a18-8dca-f5034b63d756" (UID: "80d78919-d93b-4a18-8dca-f5034b63d756"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:59.323978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.323927    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d78919-d93b-4a18-8dca-f5034b63d756-kube-api-access-rb4xj" pod "80d78919-d93b-4a18-8dca-f5034b63d756" (UID: "80d78919-d93b-4a18-8dca-f5034b63d756"). InnerVolumeSpecName "kube-api-access-rb4xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:26:59.376900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.375510    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-openshift-ca" pod "80d78919-d93b-4a18-8dca-f5034b63d756" (UID: "80d78919-d93b-4a18-8dca-f5034b63d756"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:59.415314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.415159    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:26:59.415314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.415192    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d78919-d93b-4a18-8dca-f5034b63d756-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:26:59.415314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.415208    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb4xj\" (UniqueName: \"kubernetes.io/projected/80d78919-d93b-4a18-8dca-f5034b63d756-kube-api-access-rb4xj\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:26:59.431791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.431501    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:26:59.617630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.616828    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-kube-api-access-59jfr\" (UniqueName: \"kubernetes.io/projected/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-kube-api-access-59jfr\") pod \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") "
Nov 02 01:26:59.617630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.616903    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-xks-ca\") pod \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") "
Nov 02 01:26:59.617630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.617011    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-openshift-ca\") pod \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\" (UID: \"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51\") "
Nov 02 01:26:59.618314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.618088    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-openshift-ca" pod "df2e8e49-e371-4d51-88e8-3fe8e3b3ea51" (UID: "df2e8e49-e371-4d51-88e8-3fe8e3b3ea51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:59.640754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.640644    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-kube-api-access-59jfr" pod "df2e8e49-e371-4d51-88e8-3fe8e3b3ea51" (UID: "df2e8e49-e371-4d51-88e8-3fe8e3b3ea51"). InnerVolumeSpecName "kube-api-access-59jfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:26:59.640754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.640722    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-xks-ca" pod "df2e8e49-e371-4d51-88e8-3fe8e3b3ea51" (UID: "df2e8e49-e371-4d51-88e8-3fe8e3b3ea51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:59.719111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.719075    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59jfr\" (UniqueName: \"kubernetes.io/projected/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-kube-api-access-59jfr\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:26:59.719111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.719113    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:26:59.719336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:26:59.719129    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:00.027341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:00.023485    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j" event={"ID":"df2e8e49-e371-4d51-88e8-3fe8e3b3ea51","Type":"ContainerDied","Data":"53d5a74ea3d04e2920f33c9cf34998f2a2e7484369ee1a1999d72d34c2597787"}
Nov 02 01:27:00.027341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:00.023551    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53d5a74ea3d04e2920f33c9cf34998f2a2e7484369ee1a1999d72d34c2597787"
Nov 02 01:27:00.027341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:00.024983    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"
Nov 02 01:27:00.046673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:00.043086    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk" event={"ID":"80d78919-d93b-4a18-8dca-f5034b63d756","Type":"ContainerDied","Data":"79892cd40d425ca92d58110e5c1e65b323bc5e7cb06cccb5ab41288e248b494d"}
Nov 02 01:27:00.046673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:00.043156    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79892cd40d425ca92d58110e5c1e65b323bc5e7cb06cccb5ab41288e248b494d"
Nov 02 01:27:00.046673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:00.043240    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"
Nov 02 01:27:00.138076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:00.137116    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv" event={"ID":"a53319cf-e568-412f-be8c-cd5b056ff07e","Type":"ContainerStarted","Data":"ac3cb2606798ebda5b09fa4af8629eb1694f1cb1c276d7f9aaeadcbc2d5cb109"}
Nov 02 01:27:00.138076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:00.137188    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv" event={"ID":"a53319cf-e568-412f-be8c-cd5b056ff07e","Type":"ContainerStarted","Data":"f2b98f150654ffbcc4ac156cf930a3177030180e5f51599fbb887d2a94f89dbf"}
Nov 02 01:27:00.185952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:00.185895    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv" podStartSLOduration=2.185874454 podStartE2EDuration="2.185874454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:00.184370186 +0000 UTC m=+725.489303013" watchObservedRunningTime="2026-11-02 01:27:00.185874454 +0000 UTC m=+725.490807282"
Nov 02 01:27:00.365381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:00.364211    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:01.153544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:01.153510    4851 generic.go:356] "Generic (PLEG): container finished" podID="a53319cf-e568-412f-be8c-cd5b056ff07e" containerID="ac3cb2606798ebda5b09fa4af8629eb1694f1cb1c276d7f9aaeadcbc2d5cb109" exitCode=0
Nov 02 01:27:01.153544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:01.153536    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv" event={"ID":"a53319cf-e568-412f-be8c-cd5b056ff07e","Type":"ContainerDied","Data":"ac3cb2606798ebda5b09fa4af8629eb1694f1cb1c276d7f9aaeadcbc2d5cb109"}
Nov 02 01:27:01.183596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:01.183553    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:01.930089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:01.930056    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:02.304370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.304345    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:27:02.476882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.476799    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53319cf-e568-412f-be8c-cd5b056ff07e-kube-api-access-bm27g\" (UniqueName: \"kubernetes.io/projected/a53319cf-e568-412f-be8c-cd5b056ff07e-kube-api-access-bm27g\") pod \"a53319cf-e568-412f-be8c-cd5b056ff07e\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") "
Nov 02 01:27:02.477064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.476894    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-openshift-ca\") pod \"a53319cf-e568-412f-be8c-cd5b056ff07e\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") "
Nov 02 01:27:02.477064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.477035    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-xks-ca\") pod \"a53319cf-e568-412f-be8c-cd5b056ff07e\" (UID: \"a53319cf-e568-412f-be8c-cd5b056ff07e\") "
Nov 02 01:27:02.477323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.477272    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-openshift-ca" pod "a53319cf-e568-412f-be8c-cd5b056ff07e" (UID: "a53319cf-e568-412f-be8c-cd5b056ff07e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:02.477502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.477481    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:02.479170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.479142    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53319cf-e568-412f-be8c-cd5b056ff07e-kube-api-access-bm27g" pod "a53319cf-e568-412f-be8c-cd5b056ff07e" (UID: "a53319cf-e568-412f-be8c-cd5b056ff07e"). InnerVolumeSpecName "kube-api-access-bm27g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:02.479270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.479193    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-xks-ca" pod "a53319cf-e568-412f-be8c-cd5b056ff07e" (UID: "a53319cf-e568-412f-be8c-cd5b056ff07e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:02.578122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.578093    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm27g\" (UniqueName: \"kubernetes.io/projected/a53319cf-e568-412f-be8c-cd5b056ff07e-kube-api-access-bm27g\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:02.578122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.578121    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53319cf-e568-412f-be8c-cd5b056ff07e-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:02.634607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:02.634574    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:03.167271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:03.167240    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"
Nov 02 01:27:03.167271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:03.167254    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv" event={"ID":"a53319cf-e568-412f-be8c-cd5b056ff07e","Type":"ContainerDied","Data":"f2b98f150654ffbcc4ac156cf930a3177030180e5f51599fbb887d2a94f89dbf"}
Nov 02 01:27:03.167494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:03.167337    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b98f150654ffbcc4ac156cf930a3177030180e5f51599fbb887d2a94f89dbf"
Nov 02 01:27:03.440229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:03.440107    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:04.131136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:04.131047    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:04.983594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:04.983561    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:05.268664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.268586    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"]
Nov 02 01:27:05.269123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.269108    4851 cpu_manager.go:414] "containerMap: removing container" podUID="80d78919-d93b-4a18-8dca-f5034b63d756" containerName="revoke-keys"
Nov 02 01:27:05.269183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.269126    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d78919-d93b-4a18-8dca-f5034b63d756" containerName="revoke-keys"
Nov 02 01:27:05.269183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.269138    4851 cpu_manager.go:414] "containerMap: removing container" podUID="a53319cf-e568-412f-be8c-cd5b056ff07e" containerName="revoke-keys"
Nov 02 01:27:05.269183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.269145    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53319cf-e568-412f-be8c-cd5b056ff07e" containerName="revoke-keys"
Nov 02 01:27:05.269275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.269185    4851 cpu_manager.go:414] "containerMap: removing container" podUID="df2e8e49-e371-4d51-88e8-3fe8e3b3ea51" containerName="revoke-keys"
Nov 02 01:27:05.269275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.269191    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2e8e49-e371-4d51-88e8-3fe8e3b3ea51" containerName="revoke-keys"
Nov 02 01:27:05.269275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.269267    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d78919-d93b-4a18-8dca-f5034b63d756" containerName="revoke-keys"
Nov 02 01:27:05.269390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.269276    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2e8e49-e371-4d51-88e8-3fe8e3b3ea51" containerName="revoke-keys"
Nov 02 01:27:05.269390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.269282    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53319cf-e568-412f-be8c-cd5b056ff07e" containerName="revoke-keys"
Nov 02 01:27:05.272767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.272753    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.279065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.279038    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:27:05.280817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.280514    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:27:05.280817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.280774    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:27:05.281084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.280909    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:27:05.283065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.283046    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"]
Nov 02 01:27:05.331137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.331105    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-224z4\" (UniqueName: \"kubernetes.io/projected/9645b072-7e8a-4f87-8961-704dc4b65345-kube-api-access-224z4\") pod \"maas-api-revoke-sub-dc15bc0f-ldgz8\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.331332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.331305    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc15bc0f-ldgz8\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.331442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.331426    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc15bc0f-ldgz8\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.432136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.432103    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc15bc0f-ldgz8\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.432353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.432155    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-224z4\" (UniqueName: \"kubernetes.io/projected/9645b072-7e8a-4f87-8961-704dc4b65345-kube-api-access-224z4\") pod \"maas-api-revoke-sub-dc15bc0f-ldgz8\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.432353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.432267    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc15bc0f-ldgz8\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.432935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.432906    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc15bc0f-ldgz8\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.434893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.434863    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc15bc0f-ldgz8\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.445065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.445034    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-224z4\" (UniqueName: \"kubernetes.io/projected/9645b072-7e8a-4f87-8961-704dc4b65345-kube-api-access-224z4\") pod \"maas-api-revoke-sub-dc15bc0f-ldgz8\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.588994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.588913    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:05.749891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.749815    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:05.874864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:05.874832    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"]
Nov 02 01:27:05.875850 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:05.875813    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9645b072_7e8a_4f87_8961_704dc4b65345.slice/crio-8eeeacfee375c73c9717df71efc806b146f8020577e904de8b549bc1e7108ff3 WatchSource:0}: Error finding container 8eeeacfee375c73c9717df71efc806b146f8020577e904de8b549bc1e7108ff3: Status 404 returned error can't find the container with id 8eeeacfee375c73c9717df71efc806b146f8020577e904de8b549bc1e7108ff3
Nov 02 01:27:06.190542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.190506    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8" event={"ID":"9645b072-7e8a-4f87-8961-704dc4b65345","Type":"ContainerStarted","Data":"44a720401aacaa71f069fc5364791eb4b36d203f1d4754bbb28b35c8551515de"}
Nov 02 01:27:06.190912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.190561    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8" event={"ID":"9645b072-7e8a-4f87-8961-704dc4b65345","Type":"ContainerStarted","Data":"8eeeacfee375c73c9717df71efc806b146f8020577e904de8b549bc1e7108ff3"}
Nov 02 01:27:06.210878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.210826    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8" podStartSLOduration=1.210813617 podStartE2EDuration="1.210813617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:06.20988379 +0000 UTC m=+731.514816618" watchObservedRunningTime="2026-11-02 01:27:06.210813617 +0000 UTC m=+731.515746443"
Nov 02 01:27:06.531266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.531230    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:06.737389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.737359    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"]
Nov 02 01:27:06.741270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.741244    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:06.748048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.748017    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a39108-bsjcb\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:06.748168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.748084    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krtxl\" (UniqueName: \"kubernetes.io/projected/62e8246b-bbb1-4bec-8fb5-c56925b25887-kube-api-access-krtxl\") pod \"maas-api-revoke-sub-c4a39108-bsjcb\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:06.748230 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.748207    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a39108-bsjcb\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:06.750918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.750471    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"]
Nov 02 01:27:06.849127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.849099    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a39108-bsjcb\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:06.849266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.849155    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krtxl\" (UniqueName: \"kubernetes.io/projected/62e8246b-bbb1-4bec-8fb5-c56925b25887-kube-api-access-krtxl\") pod \"maas-api-revoke-sub-c4a39108-bsjcb\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:06.849266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.849186    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a39108-bsjcb\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:06.849787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.849769    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a39108-bsjcb\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:06.851571 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.851553    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a39108-bsjcb\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:06.861689 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:06.861665    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krtxl\" (UniqueName: \"kubernetes.io/projected/62e8246b-bbb1-4bec-8fb5-c56925b25887-kube-api-access-krtxl\") pod \"maas-api-revoke-sub-c4a39108-bsjcb\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:07.116261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.116190    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:07.200330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.199508    4851 generic.go:356] "Generic (PLEG): container finished" podID="9645b072-7e8a-4f87-8961-704dc4b65345" containerID="44a720401aacaa71f069fc5364791eb4b36d203f1d4754bbb28b35c8551515de" exitCode=0
Nov 02 01:27:07.200330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.199584    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8" event={"ID":"9645b072-7e8a-4f87-8961-704dc4b65345","Type":"ContainerDied","Data":"44a720401aacaa71f069fc5364791eb4b36d203f1d4754bbb28b35c8551515de"}
Nov 02 01:27:07.233623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.233509    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:07.403913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.403878    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"]
Nov 02 01:27:07.405202 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:07.405176    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e8246b_bbb1_4bec_8fb5_c56925b25887.slice/crio-bc956815fda3c436802025a63ab39f624a04bf6530079988c122c3067666614e WatchSource:0}: Error finding container bc956815fda3c436802025a63ab39f624a04bf6530079988c122c3067666614e: Status 404 returned error can't find the container with id bc956815fda3c436802025a63ab39f624a04bf6530079988c122c3067666614e
Nov 02 01:27:07.413108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.413083    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"]
Nov 02 01:27:07.418246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.418229    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:07.427159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.427135    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"]
Nov 02 01:27:07.454030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.453928    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1442a92-pf9wx\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:07.454272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.454249    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n69lg\" (UniqueName: \"kubernetes.io/projected/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-kube-api-access-n69lg\") pod \"maas-api-revoke-sub-b1442a92-pf9wx\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:07.454428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.454339    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1442a92-pf9wx\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:07.555126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.555090    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69lg\" (UniqueName: \"kubernetes.io/projected/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-kube-api-access-n69lg\") pod \"maas-api-revoke-sub-b1442a92-pf9wx\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:07.555126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.555130    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1442a92-pf9wx\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:07.555376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.555202    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1442a92-pf9wx\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:07.555903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.555829    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1442a92-pf9wx\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:07.557468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.557446    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1442a92-pf9wx\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:07.567087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.567063    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69lg\" (UniqueName: \"kubernetes.io/projected/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-kube-api-access-n69lg\") pod \"maas-api-revoke-sub-b1442a92-pf9wx\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:07.767062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:07.767038    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:08.042218 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.042182    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"]
Nov 02 01:27:08.043696 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:08.043667    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3272a2_71ff_4a53_8c0e_3fbadf0a8fb2.slice/crio-bfed7adbab1f3282b6a926acc59b227ba6c7e4d64d9a20439647ca205f877fd5 WatchSource:0}: Error finding container bfed7adbab1f3282b6a926acc59b227ba6c7e4d64d9a20439647ca205f877fd5: Status 404 returned error can't find the container with id bfed7adbab1f3282b6a926acc59b227ba6c7e4d64d9a20439647ca205f877fd5
Nov 02 01:27:08.072651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.072621    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:08.206689 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.206655    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx" event={"ID":"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2","Type":"ContainerStarted","Data":"bfed7adbab1f3282b6a926acc59b227ba6c7e4d64d9a20439647ca205f877fd5"}
Nov 02 01:27:08.208378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.208349    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb" event={"ID":"62e8246b-bbb1-4bec-8fb5-c56925b25887","Type":"ContainerStarted","Data":"2abab2afc2a75b15953ffbc52c4d1966db9ec064b9c406575475aabb38180e55"}
Nov 02 01:27:08.208511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.208406    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb" event={"ID":"62e8246b-bbb1-4bec-8fb5-c56925b25887","Type":"ContainerStarted","Data":"bc956815fda3c436802025a63ab39f624a04bf6530079988c122c3067666614e"}
Nov 02 01:27:08.228632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.228582    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb" podStartSLOduration=2.228569313 podStartE2EDuration="2.228569313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:08.226204069 +0000 UTC m=+733.531136904" watchObservedRunningTime="2026-11-02 01:27:08.228569313 +0000 UTC m=+733.533502139"
Nov 02 01:27:08.399674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.399649    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:08.468633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.468603    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-xks-ca\") pod \"9645b072-7e8a-4f87-8961-704dc4b65345\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") "
Nov 02 01:27:08.468805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.468661    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-openshift-ca\") pod \"9645b072-7e8a-4f87-8961-704dc4b65345\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") "
Nov 02 01:27:08.468805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.468767    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9645b072-7e8a-4f87-8961-704dc4b65345-kube-api-access-224z4\" (UniqueName: \"kubernetes.io/projected/9645b072-7e8a-4f87-8961-704dc4b65345-kube-api-access-224z4\") pod \"9645b072-7e8a-4f87-8961-704dc4b65345\" (UID: \"9645b072-7e8a-4f87-8961-704dc4b65345\") "
Nov 02 01:27:08.469010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.468985    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-openshift-ca" pod "9645b072-7e8a-4f87-8961-704dc4b65345" (UID: "9645b072-7e8a-4f87-8961-704dc4b65345"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:08.469174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.469148    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:08.470826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.470804    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-xks-ca" pod "9645b072-7e8a-4f87-8961-704dc4b65345" (UID: "9645b072-7e8a-4f87-8961-704dc4b65345"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:08.470962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.470946    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9645b072-7e8a-4f87-8961-704dc4b65345-kube-api-access-224z4" pod "9645b072-7e8a-4f87-8961-704dc4b65345" (UID: "9645b072-7e8a-4f87-8961-704dc4b65345"). InnerVolumeSpecName "kube-api-access-224z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:08.570214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.570178    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-224z4\" (UniqueName: \"kubernetes.io/projected/9645b072-7e8a-4f87-8961-704dc4b65345-kube-api-access-224z4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:08.570214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.570207    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9645b072-7e8a-4f87-8961-704dc4b65345-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:08.949782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:08.949746    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:09.215408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:09.215369    4851 generic.go:356] "Generic (PLEG): container finished" podID="2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2" containerID="309131c1b6aaec0a3dda09f7d27f35f764a761ef35369f44c86a191d6adc0b75" exitCode=0
Nov 02 01:27:09.215944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:09.215456    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx" event={"ID":"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2","Type":"ContainerDied","Data":"309131c1b6aaec0a3dda09f7d27f35f764a761ef35369f44c86a191d6adc0b75"}
Nov 02 01:27:09.216965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:09.216949    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"
Nov 02 01:27:09.217101 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:09.216952    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8" event={"ID":"9645b072-7e8a-4f87-8961-704dc4b65345","Type":"ContainerDied","Data":"8eeeacfee375c73c9717df71efc806b146f8020577e904de8b549bc1e7108ff3"}
Nov 02 01:27:09.217101 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:09.217084    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eeeacfee375c73c9717df71efc806b146f8020577e904de8b549bc1e7108ff3"
Nov 02 01:27:09.218582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:09.218558    4851 generic.go:356] "Generic (PLEG): container finished" podID="62e8246b-bbb1-4bec-8fb5-c56925b25887" containerID="2abab2afc2a75b15953ffbc52c4d1966db9ec064b9c406575475aabb38180e55" exitCode=0
Nov 02 01:27:09.218660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:09.218589    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb" event={"ID":"62e8246b-bbb1-4bec-8fb5-c56925b25887","Type":"ContainerDied","Data":"2abab2afc2a75b15953ffbc52c4d1966db9ec064b9c406575475aabb38180e55"}
Nov 02 01:27:09.647603 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:09.646818    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:10.403418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.403393    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:10.442731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.442703    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:10.499761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.493679    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-kube-api-access-n69lg\" (UniqueName: \"kubernetes.io/projected/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-kube-api-access-n69lg\") pod \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") "
Nov 02 01:27:10.499761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.493727    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-xks-ca\") pod \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") "
Nov 02 01:27:10.499761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.493807    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-openshift-ca\") pod \"62e8246b-bbb1-4bec-8fb5-c56925b25887\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") "
Nov 02 01:27:10.499761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.493850    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-xks-ca\") pod \"62e8246b-bbb1-4bec-8fb5-c56925b25887\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") "
Nov 02 01:27:10.499761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.493890    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-openshift-ca\") pod \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\" (UID: \"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2\") "
Nov 02 01:27:10.499761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.494007    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e8246b-bbb1-4bec-8fb5-c56925b25887-kube-api-access-krtxl\" (UniqueName: \"kubernetes.io/projected/62e8246b-bbb1-4bec-8fb5-c56925b25887-kube-api-access-krtxl\") pod \"62e8246b-bbb1-4bec-8fb5-c56925b25887\" (UID: \"62e8246b-bbb1-4bec-8fb5-c56925b25887\") "
Nov 02 01:27:10.499761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.495207    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-openshift-ca" pod "62e8246b-bbb1-4bec-8fb5-c56925b25887" (UID: "62e8246b-bbb1-4bec-8fb5-c56925b25887"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:10.499761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.498678    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-kube-api-access-n69lg" pod "2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2" (UID: "2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2"). InnerVolumeSpecName "kube-api-access-n69lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:10.500573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.500547    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-openshift-ca" pod "2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2" (UID: "2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:10.501041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.501014    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-xks-ca" pod "2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2" (UID: "2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:10.502191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.502170    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-xks-ca" pod "62e8246b-bbb1-4bec-8fb5-c56925b25887" (UID: "62e8246b-bbb1-4bec-8fb5-c56925b25887"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:10.502603 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.502584    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e8246b-bbb1-4bec-8fb5-c56925b25887-kube-api-access-krtxl" pod "62e8246b-bbb1-4bec-8fb5-c56925b25887" (UID: "62e8246b-bbb1-4bec-8fb5-c56925b25887"). InnerVolumeSpecName "kube-api-access-krtxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:10.595216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.595186    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:10.595216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.595216    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e8246b-bbb1-4bec-8fb5-c56925b25887-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:10.595460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.595226    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:10.595460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.595234    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krtxl\" (UniqueName: \"kubernetes.io/projected/62e8246b-bbb1-4bec-8fb5-c56925b25887-kube-api-access-krtxl\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:10.595460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.595243    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n69lg\" (UniqueName: \"kubernetes.io/projected/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-kube-api-access-n69lg\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:10.595460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.595253    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:10.621219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.621188    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:10.812069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.811985    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"]
Nov 02 01:27:10.812609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.812589    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9645b072-7e8a-4f87-8961-704dc4b65345" containerName="revoke-keys"
Nov 02 01:27:10.812716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.812610    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9645b072-7e8a-4f87-8961-704dc4b65345" containerName="revoke-keys"
Nov 02 01:27:10.812716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.812643    4851 cpu_manager.go:414] "containerMap: removing container" podUID="62e8246b-bbb1-4bec-8fb5-c56925b25887" containerName="revoke-keys"
Nov 02 01:27:10.812716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.812651    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e8246b-bbb1-4bec-8fb5-c56925b25887" containerName="revoke-keys"
Nov 02 01:27:10.812716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.812671    4851 cpu_manager.go:414] "containerMap: removing container" podUID="2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2" containerName="revoke-keys"
Nov 02 01:27:10.812716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.812677    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2" containerName="revoke-keys"
Nov 02 01:27:10.812973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.812808    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9645b072-7e8a-4f87-8961-704dc4b65345" containerName="revoke-keys"
Nov 02 01:27:10.812973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.812823    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e8246b-bbb1-4bec-8fb5-c56925b25887" containerName="revoke-keys"
Nov 02 01:27:10.812973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.812834    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2" containerName="revoke-keys"
Nov 02 01:27:10.816530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.816509    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:10.824539 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.824516    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"]
Nov 02 01:27:10.897542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.897510    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8416d186-wxb9w\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:10.897705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.897572    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8416d186-wxb9w\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:10.897757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.897716    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6rnk\" (UniqueName: \"kubernetes.io/projected/d323372e-2956-4b58-9e74-dffbce95381c-kube-api-access-r6rnk\") pod \"maas-api-revoke-sub-8416d186-wxb9w\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:10.999986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:10.999946    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8416d186-wxb9w\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:11.000171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.000040    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8416d186-wxb9w\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:11.000262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.000168    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6rnk\" (UniqueName: \"kubernetes.io/projected/d323372e-2956-4b58-9e74-dffbce95381c-kube-api-access-r6rnk\") pod \"maas-api-revoke-sub-8416d186-wxb9w\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:11.001443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.001414    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8416d186-wxb9w\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:11.031314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.029520    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8416d186-wxb9w\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:11.031314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.030076    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6rnk\" (UniqueName: \"kubernetes.io/projected/d323372e-2956-4b58-9e74-dffbce95381c-kube-api-access-r6rnk\") pod \"maas-api-revoke-sub-8416d186-wxb9w\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:11.134473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.134366    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:11.238769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.237090    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx" event={"ID":"2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2","Type":"ContainerDied","Data":"bfed7adbab1f3282b6a926acc59b227ba6c7e4d64d9a20439647ca205f877fd5"}
Nov 02 01:27:11.238769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.237188    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfed7adbab1f3282b6a926acc59b227ba6c7e4d64d9a20439647ca205f877fd5"
Nov 02 01:27:11.238769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.237273    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"
Nov 02 01:27:11.246778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.246306    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb" event={"ID":"62e8246b-bbb1-4bec-8fb5-c56925b25887","Type":"ContainerDied","Data":"bc956815fda3c436802025a63ab39f624a04bf6530079988c122c3067666614e"}
Nov 02 01:27:11.246778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.246374    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc956815fda3c436802025a63ab39f624a04bf6530079988c122c3067666614e"
Nov 02 01:27:11.246778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.246462    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"
Nov 02 01:27:11.502002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:11.500194    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"]
Nov 02 01:27:11.643328 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:11.639953    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd323372e_2956_4b58_9e74_dffbce95381c.slice/crio-b246b010f8bb56993ccc3a2f6a38c5fcf15c752e8e1b391f94dcd1f87d106f85 WatchSource:0}: Error finding container b246b010f8bb56993ccc3a2f6a38c5fcf15c752e8e1b391f94dcd1f87d106f85: Status 404 returned error can't find the container with id b246b010f8bb56993ccc3a2f6a38c5fcf15c752e8e1b391f94dcd1f87d106f85
Nov 02 01:27:12.046997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.046889    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:12.307387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.306281    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w" event={"ID":"d323372e-2956-4b58-9e74-dffbce95381c","Type":"ContainerStarted","Data":"b246b010f8bb56993ccc3a2f6a38c5fcf15c752e8e1b391f94dcd1f87d106f85"}
Nov 02 01:27:12.713809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.713770    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"]
Nov 02 01:27:12.719336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.718637    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:12.743383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.734881    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"]
Nov 02 01:27:12.813204 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.812987    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27e25e0d-bqw7g\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:12.813204 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.813060    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5wp\" (UniqueName: \"kubernetes.io/projected/5f38900d-daa1-4af7-be67-1ccab966f772-kube-api-access-xm5wp\") pod \"maas-api-revoke-sub-27e25e0d-bqw7g\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:12.815493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.813334    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27e25e0d-bqw7g\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:12.917229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.917179    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27e25e0d-bqw7g\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:12.917433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.917249    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5wp\" (UniqueName: \"kubernetes.io/projected/5f38900d-daa1-4af7-be67-1ccab966f772-kube-api-access-xm5wp\") pod \"maas-api-revoke-sub-27e25e0d-bqw7g\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:12.917500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.917430    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27e25e0d-bqw7g\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:12.918513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.918488    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27e25e0d-bqw7g\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:12.922664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.922630    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27e25e0d-bqw7g\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:12.930499 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:12.930469    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5wp\" (UniqueName: \"kubernetes.io/projected/5f38900d-daa1-4af7-be67-1ccab966f772-kube-api-access-xm5wp\") pod \"maas-api-revoke-sub-27e25e0d-bqw7g\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:13.053369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:13.052809    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:13.425043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:13.422710    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:13.532202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:13.532167    4851 generic.go:356] "Generic (PLEG): container finished" podID="d323372e-2956-4b58-9e74-dffbce95381c" containerID="ba21aa30fff6834276a5229759d96dcd59e71ff5e8a42f48768c4cff275fb2a5" exitCode=0
Nov 02 01:27:13.579354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:13.532238    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w" event={"ID":"d323372e-2956-4b58-9e74-dffbce95381c","Type":"ContainerDied","Data":"ba21aa30fff6834276a5229759d96dcd59e71ff5e8a42f48768c4cff275fb2a5"}
Nov 02 01:27:13.710833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:13.710677    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"]
Nov 02 01:27:13.715151 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:13.715110    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f38900d_daa1_4af7_be67_1ccab966f772.slice/crio-7bcdcbd6b29b709452486534e9d01711bbac6a6d9a6edf4dcc4b8f9a05ed66df WatchSource:0}: Error finding container 7bcdcbd6b29b709452486534e9d01711bbac6a6d9a6edf4dcc4b8f9a05ed66df: Status 404 returned error can't find the container with id 7bcdcbd6b29b709452486534e9d01711bbac6a6d9a6edf4dcc4b8f9a05ed66df
Nov 02 01:27:14.148320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.143045    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:14.589136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.589100    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g" event={"ID":"5f38900d-daa1-4af7-be67-1ccab966f772","Type":"ContainerStarted","Data":"9ae04864daa16b05057acccf5673d4ee3eea21aa90e27640540009728ed03560"}
Nov 02 01:27:14.589351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.589156    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g" event={"ID":"5f38900d-daa1-4af7-be67-1ccab966f772","Type":"ContainerStarted","Data":"7bcdcbd6b29b709452486534e9d01711bbac6a6d9a6edf4dcc4b8f9a05ed66df"}
Nov 02 01:27:14.752759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.752726    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:14.772321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.772237    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g" podStartSLOduration=2.772218609 podStartE2EDuration="2.772218609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:14.609649926 +0000 UTC m=+739.914582754" watchObservedRunningTime="2026-11-02 01:27:14.772218609 +0000 UTC m=+740.077151436"
Nov 02 01:27:14.864512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.864488    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-openshift-ca\") pod \"d323372e-2956-4b58-9e74-dffbce95381c\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") "
Nov 02 01:27:14.864618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.864609    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-xks-ca\") pod \"d323372e-2956-4b58-9e74-dffbce95381c\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") "
Nov 02 01:27:14.864696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.864681    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d323372e-2956-4b58-9e74-dffbce95381c-kube-api-access-r6rnk\" (UniqueName: \"kubernetes.io/projected/d323372e-2956-4b58-9e74-dffbce95381c-kube-api-access-r6rnk\") pod \"d323372e-2956-4b58-9e74-dffbce95381c\" (UID: \"d323372e-2956-4b58-9e74-dffbce95381c\") "
Nov 02 01:27:14.864772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.864751    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-openshift-ca" pod "d323372e-2956-4b58-9e74-dffbce95381c" (UID: "d323372e-2956-4b58-9e74-dffbce95381c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:14.864962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.864944    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:14.866657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.866635    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-xks-ca" pod "d323372e-2956-4b58-9e74-dffbce95381c" (UID: "d323372e-2956-4b58-9e74-dffbce95381c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:14.866732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.866656    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d323372e-2956-4b58-9e74-dffbce95381c-kube-api-access-r6rnk" pod "d323372e-2956-4b58-9e74-dffbce95381c" (UID: "d323372e-2956-4b58-9e74-dffbce95381c"). InnerVolumeSpecName "kube-api-access-r6rnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:14.920560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.920534    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:14.970333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.966010    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6rnk\" (UniqueName: \"kubernetes.io/projected/d323372e-2956-4b58-9e74-dffbce95381c-kube-api-access-r6rnk\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:14.970333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:14.966040    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d323372e-2956-4b58-9e74-dffbce95381c-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:15.596706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:15.596679    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"
Nov 02 01:27:15.596897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:15.596715    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w" event={"ID":"d323372e-2956-4b58-9e74-dffbce95381c","Type":"ContainerDied","Data":"b246b010f8bb56993ccc3a2f6a38c5fcf15c752e8e1b391f94dcd1f87d106f85"}
Nov 02 01:27:15.596897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:15.596754    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b246b010f8bb56993ccc3a2f6a38c5fcf15c752e8e1b391f94dcd1f87d106f85"
Nov 02 01:27:15.598583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:15.598555    4851 generic.go:356] "Generic (PLEG): container finished" podID="5f38900d-daa1-4af7-be67-1ccab966f772" containerID="9ae04864daa16b05057acccf5673d4ee3eea21aa90e27640540009728ed03560" exitCode=0
Nov 02 01:27:15.598695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:15.598634    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g" event={"ID":"5f38900d-daa1-4af7-be67-1ccab966f772","Type":"ContainerDied","Data":"9ae04864daa16b05057acccf5673d4ee3eea21aa90e27640540009728ed03560"}
Nov 02 01:27:15.640814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:15.640785    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:16.421203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.421175    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:16.758625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.758594    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:16.786836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.786803    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-xks-ca\") pod \"5f38900d-daa1-4af7-be67-1ccab966f772\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") "
Nov 02 01:27:16.787317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.786937    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f38900d-daa1-4af7-be67-1ccab966f772-kube-api-access-xm5wp\" (UniqueName: \"kubernetes.io/projected/5f38900d-daa1-4af7-be67-1ccab966f772-kube-api-access-xm5wp\") pod \"5f38900d-daa1-4af7-be67-1ccab966f772\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") "
Nov 02 01:27:16.787317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.787028    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-openshift-ca\") pod \"5f38900d-daa1-4af7-be67-1ccab966f772\" (UID: \"5f38900d-daa1-4af7-be67-1ccab966f772\") "
Nov 02 01:27:16.787804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.787780    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-openshift-ca" pod "5f38900d-daa1-4af7-be67-1ccab966f772" (UID: "5f38900d-daa1-4af7-be67-1ccab966f772"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:16.789750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.789715    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f38900d-daa1-4af7-be67-1ccab966f772-kube-api-access-xm5wp" pod "5f38900d-daa1-4af7-be67-1ccab966f772" (UID: "5f38900d-daa1-4af7-be67-1ccab966f772"). InnerVolumeSpecName "kube-api-access-xm5wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:16.789852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.789816    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-xks-ca" pod "5f38900d-daa1-4af7-be67-1ccab966f772" (UID: "5f38900d-daa1-4af7-be67-1ccab966f772"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:16.888733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.888704    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:16.888733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.888732    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5wp\" (UniqueName: \"kubernetes.io/projected/5f38900d-daa1-4af7-be67-1ccab966f772-kube-api-access-xm5wp\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:16.888733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:16.888741    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f38900d-daa1-4af7-be67-1ccab966f772-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:17.259559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:17.259522    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:17.617751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:17.617665    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g" event={"ID":"5f38900d-daa1-4af7-be67-1ccab966f772","Type":"ContainerDied","Data":"7bcdcbd6b29b709452486534e9d01711bbac6a6d9a6edf4dcc4b8f9a05ed66df"}
Nov 02 01:27:17.618361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:17.618336    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bcdcbd6b29b709452486534e9d01711bbac6a6d9a6edf4dcc4b8f9a05ed66df"
Nov 02 01:27:17.618361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:17.617769    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"
Nov 02 01:27:17.659231 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:27:17.659184    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:27:17.705030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:17.704691    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:18.042958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:18.042863    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:18.888780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:18.888731    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:19.566610 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.566577    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"]
Nov 02 01:27:19.567093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.567067    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d323372e-2956-4b58-9e74-dffbce95381c" containerName="revoke-keys"
Nov 02 01:27:19.567093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.567083    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d323372e-2956-4b58-9e74-dffbce95381c" containerName="revoke-keys"
Nov 02 01:27:19.567093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.567093    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5f38900d-daa1-4af7-be67-1ccab966f772" containerName="revoke-keys"
Nov 02 01:27:19.567093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.567100    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f38900d-daa1-4af7-be67-1ccab966f772" containerName="revoke-keys"
Nov 02 01:27:19.567424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.567202    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f38900d-daa1-4af7-be67-1ccab966f772" containerName="revoke-keys"
Nov 02 01:27:19.567424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.567210    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d323372e-2956-4b58-9e74-dffbce95381c" containerName="revoke-keys"
Nov 02 01:27:19.570879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.570778    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.575764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.575739    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"]
Nov 02 01:27:19.578840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.578820    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:27:19.578988 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.578952    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:27:19.579067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.578997    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:27:19.579067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.579061    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:27:19.580189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.580165    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:19.584690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.584669    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"]
Nov 02 01:27:19.588844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.588822    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"]
Nov 02 01:27:19.621390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.621352    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj794\" (UniqueName: \"kubernetes.io/projected/53e5ad66-39b3-44b0-9f51-5deb62085a2b-kube-api-access-wj794\") pod \"maas-api-revoke-sub-19e19298-8jfds\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:19.621541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.621442    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ae11ef5-zsb8x\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.621541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.621483    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbnsc\" (UniqueName: \"kubernetes.io/projected/aeaa1973-6b38-4517-ac80-9670e3a42655-kube-api-access-wbnsc\") pod \"maas-api-revoke-sub-2ae11ef5-zsb8x\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.621658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.621610    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ae11ef5-zsb8x\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.621833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.621812    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19e19298-8jfds\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:19.621915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.621850    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19e19298-8jfds\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:19.649799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.649770    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:19.722887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.722856    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19e19298-8jfds\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:19.722887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.722890    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19e19298-8jfds\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:19.723116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.722954    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj794\" (UniqueName: \"kubernetes.io/projected/53e5ad66-39b3-44b0-9f51-5deb62085a2b-kube-api-access-wj794\") pod \"maas-api-revoke-sub-19e19298-8jfds\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:19.723116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.722977    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ae11ef5-zsb8x\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.723116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.722993    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbnsc\" (UniqueName: \"kubernetes.io/projected/aeaa1973-6b38-4517-ac80-9670e3a42655-kube-api-access-wbnsc\") pod \"maas-api-revoke-sub-2ae11ef5-zsb8x\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.723116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.723083    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ae11ef5-zsb8x\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.723716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.723684    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19e19298-8jfds\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:19.723991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.723759    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ae11ef5-zsb8x\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.725632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.725609    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19e19298-8jfds\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:19.725697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.725683    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ae11ef5-zsb8x\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.735069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.735045    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbnsc\" (UniqueName: \"kubernetes.io/projected/aeaa1973-6b38-4517-ac80-9670e3a42655-kube-api-access-wbnsc\") pod \"maas-api-revoke-sub-2ae11ef5-zsb8x\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.735166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.735149    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj794\" (UniqueName: \"kubernetes.io/projected/53e5ad66-39b3-44b0-9f51-5deb62085a2b-kube-api-access-wj794\") pod \"maas-api-revoke-sub-19e19298-8jfds\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:19.891685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.891592    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:19.902934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:19.902899    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:20.180494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.180465    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"]
Nov 02 01:27:20.181188 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:20.181153    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaa1973_6b38_4517_ac80_9670e3a42655.slice/crio-4d7fca3eea57612b2b2debc5721614634cadb0d9fd47b90d4e94484dde41eb43 WatchSource:0}: Error finding container 4d7fca3eea57612b2b2debc5721614634cadb0d9fd47b90d4e94484dde41eb43: Status 404 returned error can't find the container with id 4d7fca3eea57612b2b2debc5721614634cadb0d9fd47b90d4e94484dde41eb43
Nov 02 01:27:20.204215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.204191    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"]
Nov 02 01:27:20.220527 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:20.220502    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e5ad66_39b3_44b0_9f51_5deb62085a2b.slice/crio-c0800320382ccf7d0160e3074f0a3fd07ce05347eb3ebd896581bf60d0755a13 WatchSource:0}: Error finding container c0800320382ccf7d0160e3074f0a3fd07ce05347eb3ebd896581bf60d0755a13: Status 404 returned error can't find the container with id c0800320382ccf7d0160e3074f0a3fd07ce05347eb3ebd896581bf60d0755a13
Nov 02 01:27:20.544497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.544462    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:20.639309 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.639259    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x" event={"ID":"aeaa1973-6b38-4517-ac80-9670e3a42655","Type":"ContainerStarted","Data":"a3e3cd5b42f2d3e11a15c1c6798a2d6224d1cd9504d88f434e630b2c8b0e5ccb"}
Nov 02 01:27:20.639474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.639387    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x" event={"ID":"aeaa1973-6b38-4517-ac80-9670e3a42655","Type":"ContainerStarted","Data":"4d7fca3eea57612b2b2debc5721614634cadb0d9fd47b90d4e94484dde41eb43"}
Nov 02 01:27:20.640588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.640565    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds" event={"ID":"53e5ad66-39b3-44b0-9f51-5deb62085a2b","Type":"ContainerStarted","Data":"c0800320382ccf7d0160e3074f0a3fd07ce05347eb3ebd896581bf60d0755a13"}
Nov 02 01:27:20.660799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.660763    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"]
Nov 02 01:27:20.663041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.663000    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x" podStartSLOduration=1.662988104 podStartE2EDuration="1.662988104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:20.656324683 +0000 UTC m=+745.961257510" watchObservedRunningTime="2026-11-02 01:27:20.662988104 +0000 UTC m=+745.967920929"
Nov 02 01:27:20.664797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.664778    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:20.673831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.673808    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"]
Nov 02 01:27:20.745521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.745453    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93c99148-2hn7m\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:20.745637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.745530    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhkz\" (UniqueName: \"kubernetes.io/projected/1994b68d-08ae-4227-bc29-97c19358fec3-kube-api-access-9mhkz\") pod \"maas-api-revoke-sub-93c99148-2hn7m\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:20.745696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.745624    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93c99148-2hn7m\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:20.846582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.846550    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93c99148-2hn7m\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:20.846759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.846609    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhkz\" (UniqueName: \"kubernetes.io/projected/1994b68d-08ae-4227-bc29-97c19358fec3-kube-api-access-9mhkz\") pod \"maas-api-revoke-sub-93c99148-2hn7m\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:20.846759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.846648    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93c99148-2hn7m\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:20.847259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.847234    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93c99148-2hn7m\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:20.849072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.849052    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93c99148-2hn7m\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:20.858608 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.858584    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhkz\" (UniqueName: \"kubernetes.io/projected/1994b68d-08ae-4227-bc29-97c19358fec3-kube-api-access-9mhkz\") pod \"maas-api-revoke-sub-93c99148-2hn7m\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:20.980249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:20.980223    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:21.259232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:21.259207    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"]
Nov 02 01:27:21.296123 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:21.296052    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1994b68d_08ae_4227_bc29_97c19358fec3.slice/crio-ea9d8a52f59b49f26dc239fb5a8cb42605575a36ae61e9d2fc11bcb58397c38e WatchSource:0}: Error finding container ea9d8a52f59b49f26dc239fb5a8cb42605575a36ae61e9d2fc11bcb58397c38e: Status 404 returned error can't find the container with id ea9d8a52f59b49f26dc239fb5a8cb42605575a36ae61e9d2fc11bcb58397c38e
Nov 02 01:27:21.333710 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:21.333682    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:21.648358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:21.648322    4851 generic.go:356] "Generic (PLEG): container finished" podID="53e5ad66-39b3-44b0-9f51-5deb62085a2b" containerID="3ad1e364da3a53413a6ce256f7b01e70d0e0dea39fda0c0b150924f41e7b9761" exitCode=0
Nov 02 01:27:21.648573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:21.648409    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds" event={"ID":"53e5ad66-39b3-44b0-9f51-5deb62085a2b","Type":"ContainerDied","Data":"3ad1e364da3a53413a6ce256f7b01e70d0e0dea39fda0c0b150924f41e7b9761"}
Nov 02 01:27:21.650050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:21.650024    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m" event={"ID":"1994b68d-08ae-4227-bc29-97c19358fec3","Type":"ContainerStarted","Data":"455bea78cc04bfc96f915db9d3252f3c68377274213a2d50ff4a32400321edb0"}
Nov 02 01:27:21.650159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:21.650071    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m" event={"ID":"1994b68d-08ae-4227-bc29-97c19358fec3","Type":"ContainerStarted","Data":"ea9d8a52f59b49f26dc239fb5a8cb42605575a36ae61e9d2fc11bcb58397c38e"}
Nov 02 01:27:21.651649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:21.651630    4851 generic.go:356] "Generic (PLEG): container finished" podID="aeaa1973-6b38-4517-ac80-9670e3a42655" containerID="a3e3cd5b42f2d3e11a15c1c6798a2d6224d1cd9504d88f434e630b2c8b0e5ccb" exitCode=0
Nov 02 01:27:21.651740 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:21.651709    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x" event={"ID":"aeaa1973-6b38-4517-ac80-9670e3a42655","Type":"ContainerDied","Data":"a3e3cd5b42f2d3e11a15c1c6798a2d6224d1cd9504d88f434e630b2c8b0e5ccb"}
Nov 02 01:27:21.689592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:21.689537    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m" podStartSLOduration=1.6895248619999998 podStartE2EDuration="1.689524862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:21.68582654 +0000 UTC m=+746.990759365" watchObservedRunningTime="2026-11-02 01:27:21.689524862 +0000 UTC m=+746.994457688"
Nov 02 01:27:22.028109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.028059    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:22.659769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.659740    4851 generic.go:356] "Generic (PLEG): container finished" podID="1994b68d-08ae-4227-bc29-97c19358fec3" containerID="455bea78cc04bfc96f915db9d3252f3c68377274213a2d50ff4a32400321edb0" exitCode=0
Nov 02 01:27:22.659933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.659832    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m" event={"ID":"1994b68d-08ae-4227-bc29-97c19358fec3","Type":"ContainerDied","Data":"455bea78cc04bfc96f915db9d3252f3c68377274213a2d50ff4a32400321edb0"}
Nov 02 01:27:22.665020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.664995    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:22.793241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.793200    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:22.843793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.843767    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:22.878737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.878715    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:22.941319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.931446    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:22.972451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.972425    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-openshift-ca\") pod \"aeaa1973-6b38-4517-ac80-9670e3a42655\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") "
Nov 02 01:27:22.972593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.972458    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-xks-ca\") pod \"aeaa1973-6b38-4517-ac80-9670e3a42655\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") "
Nov 02 01:27:22.972593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.972515    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeaa1973-6b38-4517-ac80-9670e3a42655-kube-api-access-wbnsc\" (UniqueName: \"kubernetes.io/projected/aeaa1973-6b38-4517-ac80-9670e3a42655-kube-api-access-wbnsc\") pod \"aeaa1973-6b38-4517-ac80-9670e3a42655\" (UID: \"aeaa1973-6b38-4517-ac80-9670e3a42655\") "
Nov 02 01:27:22.972711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.972594    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-xks-ca\") pod \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") "
Nov 02 01:27:22.972711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.972660    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-openshift-ca\") pod \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") "
Nov 02 01:27:22.972711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.972695    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e5ad66-39b3-44b0-9f51-5deb62085a2b-kube-api-access-wj794\" (UniqueName: \"kubernetes.io/projected/53e5ad66-39b3-44b0-9f51-5deb62085a2b-kube-api-access-wj794\") pod \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\" (UID: \"53e5ad66-39b3-44b0-9f51-5deb62085a2b\") "
Nov 02 01:27:22.972870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.972832    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-openshift-ca" pod "aeaa1973-6b38-4517-ac80-9670e3a42655" (UID: "aeaa1973-6b38-4517-ac80-9670e3a42655"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:22.973060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.973035    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-openshift-ca" pod "53e5ad66-39b3-44b0-9f51-5deb62085a2b" (UID: "53e5ad66-39b3-44b0-9f51-5deb62085a2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:22.973469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.973281    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:22.973469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.973412    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:22.975065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.975036    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-xks-ca" pod "aeaa1973-6b38-4517-ac80-9670e3a42655" (UID: "aeaa1973-6b38-4517-ac80-9670e3a42655"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:22.975065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.975043    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-xks-ca" pod "53e5ad66-39b3-44b0-9f51-5deb62085a2b" (UID: "53e5ad66-39b3-44b0-9f51-5deb62085a2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:22.975065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.975056    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e5ad66-39b3-44b0-9f51-5deb62085a2b-kube-api-access-wj794" pod "53e5ad66-39b3-44b0-9f51-5deb62085a2b" (UID: "53e5ad66-39b3-44b0-9f51-5deb62085a2b"). InnerVolumeSpecName "kube-api-access-wj794". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:22.975248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:22.975097    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeaa1973-6b38-4517-ac80-9670e3a42655-kube-api-access-wbnsc" pod "aeaa1973-6b38-4517-ac80-9670e3a42655" (UID: "aeaa1973-6b38-4517-ac80-9670e3a42655"). InnerVolumeSpecName "kube-api-access-wbnsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:23.074341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.074315    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbnsc\" (UniqueName: \"kubernetes.io/projected/aeaa1973-6b38-4517-ac80-9670e3a42655-kube-api-access-wbnsc\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:23.074341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.074343    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e5ad66-39b3-44b0-9f51-5deb62085a2b-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:23.074716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.074356    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj794\" (UniqueName: \"kubernetes.io/projected/53e5ad66-39b3-44b0-9f51-5deb62085a2b-kube-api-access-wj794\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:23.074716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.074369    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeaa1973-6b38-4517-ac80-9670e3a42655-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:23.117346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.117317    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"]
Nov 02 01:27:23.117854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.117841    4851 cpu_manager.go:414] "containerMap: removing container" podUID="53e5ad66-39b3-44b0-9f51-5deb62085a2b" containerName="revoke-keys"
Nov 02 01:27:23.117891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.117856    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e5ad66-39b3-44b0-9f51-5deb62085a2b" containerName="revoke-keys"
Nov 02 01:27:23.117891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.117888    4851 cpu_manager.go:414] "containerMap: removing container" podUID="aeaa1973-6b38-4517-ac80-9670e3a42655" containerName="revoke-keys"
Nov 02 01:27:23.117968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.117893    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeaa1973-6b38-4517-ac80-9670e3a42655" containerName="revoke-keys"
Nov 02 01:27:23.117968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.117966    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeaa1973-6b38-4517-ac80-9670e3a42655" containerName="revoke-keys"
Nov 02 01:27:23.118026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.117978    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e5ad66-39b3-44b0-9f51-5deb62085a2b" containerName="revoke-keys"
Nov 02 01:27:23.121390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.121372    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.130043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.130019    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"]
Nov 02 01:27:23.175556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.175518    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f32e8a0a-2j8nz\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.175688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.175563    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnflv\" (UniqueName: \"kubernetes.io/projected/8919b179-eceb-4b01-9a7a-2a56983be518-kube-api-access-cnflv\") pod \"maas-api-revoke-sub-f32e8a0a-2j8nz\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.175752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.175686    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f32e8a0a-2j8nz\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.276927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.276834    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f32e8a0a-2j8nz\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.276927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.276871    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnflv\" (UniqueName: \"kubernetes.io/projected/8919b179-eceb-4b01-9a7a-2a56983be518-kube-api-access-cnflv\") pod \"maas-api-revoke-sub-f32e8a0a-2j8nz\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.277155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.276929    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f32e8a0a-2j8nz\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.277829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.277808    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f32e8a0a-2j8nz\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.279369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.279348    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f32e8a0a-2j8nz\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.288898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.288877    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnflv\" (UniqueName: \"kubernetes.io/projected/8919b179-eceb-4b01-9a7a-2a56983be518-kube-api-access-cnflv\") pod \"maas-api-revoke-sub-f32e8a0a-2j8nz\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.436854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.436819    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:23.632458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.632385    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:23.688579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.688541    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x" event={"ID":"aeaa1973-6b38-4517-ac80-9670e3a42655","Type":"ContainerDied","Data":"4d7fca3eea57612b2b2debc5721614634cadb0d9fd47b90d4e94484dde41eb43"}
Nov 02 01:27:23.688761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.688622    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d7fca3eea57612b2b2debc5721614634cadb0d9fd47b90d4e94484dde41eb43"
Nov 02 01:27:23.688761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.688692    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"
Nov 02 01:27:23.694346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.694278    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"
Nov 02 01:27:23.694494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.694305    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds" event={"ID":"53e5ad66-39b3-44b0-9f51-5deb62085a2b","Type":"ContainerDied","Data":"c0800320382ccf7d0160e3074f0a3fd07ce05347eb3ebd896581bf60d0755a13"}
Nov 02 01:27:23.694494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.694446    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0800320382ccf7d0160e3074f0a3fd07ce05347eb3ebd896581bf60d0755a13"
Nov 02 01:27:23.737930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.737902    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"]
Nov 02 01:27:23.739135 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:23.739111    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8919b179_eceb_4b01_9a7a_2a56983be518.slice/crio-2671213696f66ac41691b845f5faa369f448579186778d0e0d9faa76f1dfed13 WatchSource:0}: Error finding container 2671213696f66ac41691b845f5faa369f448579186778d0e0d9faa76f1dfed13: Status 404 returned error can't find the container with id 2671213696f66ac41691b845f5faa369f448579186778d0e0d9faa76f1dfed13
Nov 02 01:27:23.839042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.839015    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"]
Nov 02 01:27:23.844174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.844156    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:23.852142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.852117    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"]
Nov 02 01:27:23.858178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.858154    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:23.889746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.889670    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e24150c6-xcvnh\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:23.889864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.889831    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm6wg\" (UniqueName: \"kubernetes.io/projected/42bf9d27-9ccc-43cf-bc68-9e6987378520-kube-api-access-zm6wg\") pod \"maas-api-revoke-sub-e24150c6-xcvnh\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:23.889968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.889954    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e24150c6-xcvnh\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:23.991197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.991170    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-openshift-ca\") pod \"1994b68d-08ae-4227-bc29-97c19358fec3\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") "
Nov 02 01:27:23.991402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.991269    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1994b68d-08ae-4227-bc29-97c19358fec3-kube-api-access-9mhkz\" (UniqueName: \"kubernetes.io/projected/1994b68d-08ae-4227-bc29-97c19358fec3-kube-api-access-9mhkz\") pod \"1994b68d-08ae-4227-bc29-97c19358fec3\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") "
Nov 02 01:27:23.991402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.991367    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-xks-ca\") pod \"1994b68d-08ae-4227-bc29-97c19358fec3\" (UID: \"1994b68d-08ae-4227-bc29-97c19358fec3\") "
Nov 02 01:27:23.991654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.991615    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-openshift-ca" pod "1994b68d-08ae-4227-bc29-97c19358fec3" (UID: "1994b68d-08ae-4227-bc29-97c19358fec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:23.991654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.991636    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm6wg\" (UniqueName: \"kubernetes.io/projected/42bf9d27-9ccc-43cf-bc68-9e6987378520-kube-api-access-zm6wg\") pod \"maas-api-revoke-sub-e24150c6-xcvnh\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:23.991788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.991762    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e24150c6-xcvnh\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:23.991860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.991846    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e24150c6-xcvnh\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:23.992004 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.991990    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:23.992790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.992762    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e24150c6-xcvnh\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:23.993804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.993778    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-xks-ca" pod "1994b68d-08ae-4227-bc29-97c19358fec3" (UID: "1994b68d-08ae-4227-bc29-97c19358fec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:23.993918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.993831    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1994b68d-08ae-4227-bc29-97c19358fec3-kube-api-access-9mhkz" pod "1994b68d-08ae-4227-bc29-97c19358fec3" (UID: "1994b68d-08ae-4227-bc29-97c19358fec3"). InnerVolumeSpecName "kube-api-access-9mhkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:23.994279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:23.994264    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e24150c6-xcvnh\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:24.007558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.007536    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm6wg\" (UniqueName: \"kubernetes.io/projected/42bf9d27-9ccc-43cf-bc68-9e6987378520-kube-api-access-zm6wg\") pod \"maas-api-revoke-sub-e24150c6-xcvnh\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:24.093622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.093589    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mhkz\" (UniqueName: \"kubernetes.io/projected/1994b68d-08ae-4227-bc29-97c19358fec3-kube-api-access-9mhkz\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:24.093955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.093626    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1994b68d-08ae-4227-bc29-97c19358fec3-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:24.167453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.167423    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:24.445213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.445184    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"]
Nov 02 01:27:24.445751 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:24.445725    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42bf9d27_9ccc_43cf_bc68_9e6987378520.slice/crio-ffb96b0de0ec8d8d39dc1d39421f0caa13a8c50646addd96eb6faa1e5c055705 WatchSource:0}: Error finding container ffb96b0de0ec8d8d39dc1d39421f0caa13a8c50646addd96eb6faa1e5c055705: Status 404 returned error can't find the container with id ffb96b0de0ec8d8d39dc1d39421f0caa13a8c50646addd96eb6faa1e5c055705
Nov 02 01:27:24.638656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.638621    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:24.704359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.704262    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"
Nov 02 01:27:24.704359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.704258    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m" event={"ID":"1994b68d-08ae-4227-bc29-97c19358fec3","Type":"ContainerDied","Data":"ea9d8a52f59b49f26dc239fb5a8cb42605575a36ae61e9d2fc11bcb58397c38e"}
Nov 02 01:27:24.704581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.704392    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea9d8a52f59b49f26dc239fb5a8cb42605575a36ae61e9d2fc11bcb58397c38e"
Nov 02 01:27:24.706121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.706093    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz" event={"ID":"8919b179-eceb-4b01-9a7a-2a56983be518","Type":"ContainerStarted","Data":"4ed0e21e866e8dd1ed482ee26c55c33af3269feea86afb89e8d0aabaced167f7"}
Nov 02 01:27:24.706257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.706141    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz" event={"ID":"8919b179-eceb-4b01-9a7a-2a56983be518","Type":"ContainerStarted","Data":"2671213696f66ac41691b845f5faa369f448579186778d0e0d9faa76f1dfed13"}
Nov 02 01:27:24.707614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.707588    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh" event={"ID":"42bf9d27-9ccc-43cf-bc68-9e6987378520","Type":"ContainerStarted","Data":"ffb96b0de0ec8d8d39dc1d39421f0caa13a8c50646addd96eb6faa1e5c055705"}
Nov 02 01:27:24.726015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:24.725959    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz" podStartSLOduration=1.725947838 podStartE2EDuration="1.725947838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:24.724018164 +0000 UTC m=+750.028950990" watchObservedRunningTime="2026-11-02 01:27:24.725947838 +0000 UTC m=+750.030880706"
Nov 02 01:27:25.745265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:25.745224    4851 generic.go:356] "Generic (PLEG): container finished" podID="8919b179-eceb-4b01-9a7a-2a56983be518" containerID="4ed0e21e866e8dd1ed482ee26c55c33af3269feea86afb89e8d0aabaced167f7" exitCode=0
Nov 02 01:27:25.745741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:25.745376    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz" event={"ID":"8919b179-eceb-4b01-9a7a-2a56983be518","Type":"ContainerDied","Data":"4ed0e21e866e8dd1ed482ee26c55c33af3269feea86afb89e8d0aabaced167f7"}
Nov 02 01:27:25.748023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:25.747998    4851 generic.go:356] "Generic (PLEG): container finished" podID="42bf9d27-9ccc-43cf-bc68-9e6987378520" containerID="499c1f1a2cfdd76b96349214795f15b76a42d25e7bd22038ca937c5110d6ec53" exitCode=0
Nov 02 01:27:25.748210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:25.748047    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh" event={"ID":"42bf9d27-9ccc-43cf-bc68-9e6987378520","Type":"ContainerDied","Data":"499c1f1a2cfdd76b96349214795f15b76a42d25e7bd22038ca937c5110d6ec53"}
Nov 02 01:27:25.750807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:25.750776    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:26.827804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:26.827771    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:27.156099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.156063    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:27.261208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.261171    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:27.268870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.268838    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:27.356509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.356423    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-xks-ca\") pod \"42bf9d27-9ccc-43cf-bc68-9e6987378520\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") "
Nov 02 01:27:27.356509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.356467    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-openshift-ca\") pod \"42bf9d27-9ccc-43cf-bc68-9e6987378520\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") "
Nov 02 01:27:27.356691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.356527    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42bf9d27-9ccc-43cf-bc68-9e6987378520-kube-api-access-zm6wg\" (UniqueName: \"kubernetes.io/projected/42bf9d27-9ccc-43cf-bc68-9e6987378520-kube-api-access-zm6wg\") pod \"42bf9d27-9ccc-43cf-bc68-9e6987378520\" (UID: \"42bf9d27-9ccc-43cf-bc68-9e6987378520\") "
Nov 02 01:27:27.356691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.356670    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-xks-ca\") pod \"8919b179-eceb-4b01-9a7a-2a56983be518\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") "
Nov 02 01:27:27.356793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.356726    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-openshift-ca\") pod \"8919b179-eceb-4b01-9a7a-2a56983be518\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") "
Nov 02 01:27:27.356793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.356764    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8919b179-eceb-4b01-9a7a-2a56983be518-kube-api-access-cnflv\" (UniqueName: \"kubernetes.io/projected/8919b179-eceb-4b01-9a7a-2a56983be518-kube-api-access-cnflv\") pod \"8919b179-eceb-4b01-9a7a-2a56983be518\" (UID: \"8919b179-eceb-4b01-9a7a-2a56983be518\") "
Nov 02 01:27:27.356939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.356918    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-openshift-ca" pod "42bf9d27-9ccc-43cf-bc68-9e6987378520" (UID: "42bf9d27-9ccc-43cf-bc68-9e6987378520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:27.357135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.357104    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-openshift-ca" pod "8919b179-eceb-4b01-9a7a-2a56983be518" (UID: "8919b179-eceb-4b01-9a7a-2a56983be518"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:27.357239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.357226    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:27.357314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.357246    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:27.359039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.359019    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-xks-ca" pod "8919b179-eceb-4b01-9a7a-2a56983be518" (UID: "8919b179-eceb-4b01-9a7a-2a56983be518"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:27.359138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.359060    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bf9d27-9ccc-43cf-bc68-9e6987378520-kube-api-access-zm6wg" pod "42bf9d27-9ccc-43cf-bc68-9e6987378520" (UID: "42bf9d27-9ccc-43cf-bc68-9e6987378520"). InnerVolumeSpecName "kube-api-access-zm6wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:27.359211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.359197    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-xks-ca" pod "42bf9d27-9ccc-43cf-bc68-9e6987378520" (UID: "42bf9d27-9ccc-43cf-bc68-9e6987378520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:27.359412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.359396    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8919b179-eceb-4b01-9a7a-2a56983be518-kube-api-access-cnflv" pod "8919b179-eceb-4b01-9a7a-2a56983be518" (UID: "8919b179-eceb-4b01-9a7a-2a56983be518"). InnerVolumeSpecName "kube-api-access-cnflv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:27.458771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.458724    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8919b179-eceb-4b01-9a7a-2a56983be518-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:27.458771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.458761    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnflv\" (UniqueName: \"kubernetes.io/projected/8919b179-eceb-4b01-9a7a-2a56983be518-kube-api-access-cnflv\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:27.458771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.458776    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42bf9d27-9ccc-43cf-bc68-9e6987378520-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:27.459029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.458789    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm6wg\" (UniqueName: \"kubernetes.io/projected/42bf9d27-9ccc-43cf-bc68-9e6987378520-kube-api-access-zm6wg\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:27.786226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.786198    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"
Nov 02 01:27:27.786445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.786238    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh" event={"ID":"42bf9d27-9ccc-43cf-bc68-9e6987378520","Type":"ContainerDied","Data":"ffb96b0de0ec8d8d39dc1d39421f0caa13a8c50646addd96eb6faa1e5c055705"}
Nov 02 01:27:27.786445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.786310    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffb96b0de0ec8d8d39dc1d39421f0caa13a8c50646addd96eb6faa1e5c055705"
Nov 02 01:27:27.787798 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.787778    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz" event={"ID":"8919b179-eceb-4b01-9a7a-2a56983be518","Type":"ContainerDied","Data":"2671213696f66ac41691b845f5faa369f448579186778d0e0d9faa76f1dfed13"}
Nov 02 01:27:27.787893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.787799    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"
Nov 02 01:27:27.787893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:27.787817    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2671213696f66ac41691b845f5faa369f448579186778d0e0d9faa76f1dfed13"
Nov 02 01:27:28.086414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:28.074722    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:28.731349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:28.731314    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:29.434549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:29.434517    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:30.192750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.192718    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:30.857023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.856991    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"]
Nov 02 01:27:30.857561 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.857545    4851 cpu_manager.go:414] "containerMap: removing container" podUID="42bf9d27-9ccc-43cf-bc68-9e6987378520" containerName="revoke-keys"
Nov 02 01:27:30.857619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.857563    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42bf9d27-9ccc-43cf-bc68-9e6987378520" containerName="revoke-keys"
Nov 02 01:27:30.857619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.857578    4851 cpu_manager.go:414] "containerMap: removing container" podUID="1994b68d-08ae-4227-bc29-97c19358fec3" containerName="revoke-keys"
Nov 02 01:27:30.857619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.857583    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1994b68d-08ae-4227-bc29-97c19358fec3" containerName="revoke-keys"
Nov 02 01:27:30.857619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.857589    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8919b179-eceb-4b01-9a7a-2a56983be518" containerName="revoke-keys"
Nov 02 01:27:30.857619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.857593    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8919b179-eceb-4b01-9a7a-2a56983be518" containerName="revoke-keys"
Nov 02 01:27:30.857766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.857678    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8919b179-eceb-4b01-9a7a-2a56983be518" containerName="revoke-keys"
Nov 02 01:27:30.857766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.857687    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="42bf9d27-9ccc-43cf-bc68-9e6987378520" containerName="revoke-keys"
Nov 02 01:27:30.857766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.857695    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="1994b68d-08ae-4227-bc29-97c19358fec3" containerName="revoke-keys"
Nov 02 01:27:30.865217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.865197    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:30.869102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.869076    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:27:30.870062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.870042    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:27:30.870352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.870331    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:27:30.870352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.870337    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:27:30.871357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:30.871337    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"]
Nov 02 01:27:31.013610 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.013582    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj55g\" (UniqueName: \"kubernetes.io/projected/b5ed7c1d-eab3-421c-940e-328af9afc7f6-kube-api-access-xj55g\") pod \"maas-api-revoke-sub-88d85c94-jsfp2\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:31.013793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.013617    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d85c94-jsfp2\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:31.013793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.013750    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d85c94-jsfp2\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:31.115429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.115339    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj55g\" (UniqueName: \"kubernetes.io/projected/b5ed7c1d-eab3-421c-940e-328af9afc7f6-kube-api-access-xj55g\") pod \"maas-api-revoke-sub-88d85c94-jsfp2\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:31.115429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.115394    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d85c94-jsfp2\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:31.115644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.115492    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d85c94-jsfp2\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:31.116155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.116130    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d85c94-jsfp2\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:31.118088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.118067    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d85c94-jsfp2\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:31.122318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.122260    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:31.127331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.127307    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj55g\" (UniqueName: \"kubernetes.io/projected/b5ed7c1d-eab3-421c-940e-328af9afc7f6-kube-api-access-xj55g\") pod \"maas-api-revoke-sub-88d85c94-jsfp2\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:31.183319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.183252    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:31.463926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.463896    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"]
Nov 02 01:27:31.464683 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:31.464658    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ed7c1d_eab3_421c_940e_328af9afc7f6.slice/crio-1805a7ced84b764bab748a97c954c8a092fb801d8a5d5e5e7633adc830aeb801 WatchSource:0}: Error finding container 1805a7ced84b764bab748a97c954c8a092fb801d8a5d5e5e7633adc830aeb801: Status 404 returned error can't find the container with id 1805a7ced84b764bab748a97c954c8a092fb801d8a5d5e5e7633adc830aeb801
Nov 02 01:27:31.822951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.822918    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2" event={"ID":"b5ed7c1d-eab3-421c-940e-328af9afc7f6","Type":"ContainerStarted","Data":"9472bfb0e779f827120e3a8121c1009cf904e37d9420b8eedea2e81dfabed481"}
Nov 02 01:27:31.823142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.822982    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2" event={"ID":"b5ed7c1d-eab3-421c-940e-328af9afc7f6","Type":"ContainerStarted","Data":"1805a7ced84b764bab748a97c954c8a092fb801d8a5d5e5e7633adc830aeb801"}
Nov 02 01:27:31.843620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.843591    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:31.845587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:31.845543    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2" podStartSLOduration=1.845529987 podStartE2EDuration="1.845529987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:31.841361524 +0000 UTC m=+757.146294342" watchObservedRunningTime="2026-11-02 01:27:31.845529987 +0000 UTC m=+757.150462813"
Nov 02 01:27:32.282398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.282367    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"]
Nov 02 01:27:32.286333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.286316    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.293519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.293496    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"]
Nov 02 01:27:32.429671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.429636    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7aec4bf-x92wk\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.429857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.429778    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpmj\" (UniqueName: \"kubernetes.io/projected/339aa318-ced7-44f5-929c-09fa6493e1fd-kube-api-access-fqpmj\") pod \"maas-api-revoke-sub-c7aec4bf-x92wk\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.429857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.429804    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7aec4bf-x92wk\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.532995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.532903    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpmj\" (UniqueName: \"kubernetes.io/projected/339aa318-ced7-44f5-929c-09fa6493e1fd-kube-api-access-fqpmj\") pod \"maas-api-revoke-sub-c7aec4bf-x92wk\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.532995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.532947    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7aec4bf-x92wk\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.533211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.533109    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7aec4bf-x92wk\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.533932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.533887    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7aec4bf-x92wk\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.535747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.535726    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7aec4bf-x92wk\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.548157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.548130    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpmj\" (UniqueName: \"kubernetes.io/projected/339aa318-ced7-44f5-929c-09fa6493e1fd-kube-api-access-fqpmj\") pod \"maas-api-revoke-sub-c7aec4bf-x92wk\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.580975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.580950    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:32.602649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.602623    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:32.710953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.710922    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:32.861572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.861484    4851 generic.go:356] "Generic (PLEG): container finished" podID="b5ed7c1d-eab3-421c-940e-328af9afc7f6" containerID="9472bfb0e779f827120e3a8121c1009cf904e37d9420b8eedea2e81dfabed481" exitCode=0
Nov 02 01:27:32.863334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.863262    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2" event={"ID":"b5ed7c1d-eab3-421c-940e-328af9afc7f6","Type":"ContainerDied","Data":"9472bfb0e779f827120e3a8121c1009cf904e37d9420b8eedea2e81dfabed481"}
Nov 02 01:27:32.898559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:32.898528    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"]
Nov 02 01:27:32.899729 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:32.899709    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339aa318_ced7_44f5_929c_09fa6493e1fd.slice/crio-13b67269bdd94dcddabe28c88aab8c8e564ee7b74447d4cc3bd2a5ecb33e1f08 WatchSource:0}: Error finding container 13b67269bdd94dcddabe28c88aab8c8e564ee7b74447d4cc3bd2a5ecb33e1f08: Status 404 returned error can't find the container with id 13b67269bdd94dcddabe28c88aab8c8e564ee7b74447d4cc3bd2a5ecb33e1f08
Nov 02 01:27:33.493946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:33.493907    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:33.869527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:33.869494    4851 generic.go:356] "Generic (PLEG): container finished" podID="339aa318-ced7-44f5-929c-09fa6493e1fd" containerID="e54503540621365c6ca6c3dd5d9dd81bf2090accc63ec03bf90194bcbe22b724" exitCode=0
Nov 02 01:27:33.869674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:33.869585    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk" event={"ID":"339aa318-ced7-44f5-929c-09fa6493e1fd","Type":"ContainerDied","Data":"e54503540621365c6ca6c3dd5d9dd81bf2090accc63ec03bf90194bcbe22b724"}
Nov 02 01:27:33.869674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:33.869637    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk" event={"ID":"339aa318-ced7-44f5-929c-09fa6493e1fd","Type":"ContainerStarted","Data":"13b67269bdd94dcddabe28c88aab8c8e564ee7b74447d4cc3bd2a5ecb33e1f08"}
Nov 02 01:27:34.028967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.028940    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:34.137767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.137732    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"]
Nov 02 01:27:34.138237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.138224    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b5ed7c1d-eab3-421c-940e-328af9afc7f6" containerName="revoke-keys"
Nov 02 01:27:34.138282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.138238    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ed7c1d-eab3-421c-940e-328af9afc7f6" containerName="revoke-keys"
Nov 02 01:27:34.138361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.138350    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ed7c1d-eab3-421c-940e-328af9afc7f6" containerName="revoke-keys"
Nov 02 01:27:34.141621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.141606    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.150554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.150530    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"]
Nov 02 01:27:34.181970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.181944    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ed7c1d-eab3-421c-940e-328af9afc7f6-kube-api-access-xj55g\" (UniqueName: \"kubernetes.io/projected/b5ed7c1d-eab3-421c-940e-328af9afc7f6-kube-api-access-xj55g\") pod \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") "
Nov 02 01:27:34.182143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.182041    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-xks-ca\") pod \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") "
Nov 02 01:27:34.182143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.182096    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-openshift-ca\") pod \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\" (UID: \"b5ed7c1d-eab3-421c-940e-328af9afc7f6\") "
Nov 02 01:27:34.182562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.182533    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-openshift-ca" pod "b5ed7c1d-eab3-421c-940e-328af9afc7f6" (UID: "b5ed7c1d-eab3-421c-940e-328af9afc7f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:34.182845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.182662    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:34.184372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.184342    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ed7c1d-eab3-421c-940e-328af9afc7f6-kube-api-access-xj55g" pod "b5ed7c1d-eab3-421c-940e-328af9afc7f6" (UID: "b5ed7c1d-eab3-421c-940e-328af9afc7f6"). InnerVolumeSpecName "kube-api-access-xj55g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:34.184488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.184407    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-xks-ca" pod "b5ed7c1d-eab3-421c-940e-328af9afc7f6" (UID: "b5ed7c1d-eab3-421c-940e-328af9afc7f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:34.283215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.283139    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e582780-vp7p8\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.283371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.283350    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e582780-vp7p8\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.283432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.283384    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jsm\" (UniqueName: \"kubernetes.io/projected/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-kube-api-access-99jsm\") pod \"maas-api-revoke-sub-8e582780-vp7p8\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.283478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.283461    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj55g\" (UniqueName: \"kubernetes.io/projected/b5ed7c1d-eab3-421c-940e-328af9afc7f6-kube-api-access-xj55g\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:34.283478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.283473    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5ed7c1d-eab3-421c-940e-328af9afc7f6-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:34.384791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.384756    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e582780-vp7p8\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.384964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.384800    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jsm\" (UniqueName: \"kubernetes.io/projected/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-kube-api-access-99jsm\") pod \"maas-api-revoke-sub-8e582780-vp7p8\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.384964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.384862    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e582780-vp7p8\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.385501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.385478    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e582780-vp7p8\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.387212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.387191    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e582780-vp7p8\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.398738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.398712    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jsm\" (UniqueName: \"kubernetes.io/projected/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-kube-api-access-99jsm\") pod \"maas-api-revoke-sub-8e582780-vp7p8\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.421442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.421418    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:34.457890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.457771    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:34.746566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.746538    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"]
Nov 02 01:27:34.747680 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:34.747655    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ebaa34_a6ca_439e_aaf1_cf3e6564a8e6.slice/crio-cf0c271756eaa39ef6c31d26557e0a5d8a63f7c5a4bed7e19e345eefe4329df1 WatchSource:0}: Error finding container cf0c271756eaa39ef6c31d26557e0a5d8a63f7c5a4bed7e19e345eefe4329df1: Status 404 returned error can't find the container with id cf0c271756eaa39ef6c31d26557e0a5d8a63f7c5a4bed7e19e345eefe4329df1
Nov 02 01:27:34.876063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.876044    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"
Nov 02 01:27:34.876063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.876050    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2" event={"ID":"b5ed7c1d-eab3-421c-940e-328af9afc7f6","Type":"ContainerDied","Data":"1805a7ced84b764bab748a97c954c8a092fb801d8a5d5e5e7633adc830aeb801"}
Nov 02 01:27:34.876321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.876092    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1805a7ced84b764bab748a97c954c8a092fb801d8a5d5e5e7633adc830aeb801"
Nov 02 01:27:34.877507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:34.877486    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8" event={"ID":"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6","Type":"ContainerStarted","Data":"cf0c271756eaa39ef6c31d26557e0a5d8a63f7c5a4bed7e19e345eefe4329df1"}
Nov 02 01:27:35.022066 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.022044    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:35.193977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.193749    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-openshift-ca\") pod \"339aa318-ced7-44f5-929c-09fa6493e1fd\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") "
Nov 02 01:27:35.194154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.194002    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-xks-ca\") pod \"339aa318-ced7-44f5-929c-09fa6493e1fd\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") "
Nov 02 01:27:35.194154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.194044    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339aa318-ced7-44f5-929c-09fa6493e1fd-kube-api-access-fqpmj\" (UniqueName: \"kubernetes.io/projected/339aa318-ced7-44f5-929c-09fa6493e1fd-kube-api-access-fqpmj\") pod \"339aa318-ced7-44f5-929c-09fa6493e1fd\" (UID: \"339aa318-ced7-44f5-929c-09fa6493e1fd\") "
Nov 02 01:27:35.194154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.194097    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-openshift-ca" pod "339aa318-ced7-44f5-929c-09fa6493e1fd" (UID: "339aa318-ced7-44f5-929c-09fa6493e1fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:35.194537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.194488    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:35.196483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.196458    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-xks-ca" pod "339aa318-ced7-44f5-929c-09fa6493e1fd" (UID: "339aa318-ced7-44f5-929c-09fa6493e1fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:35.196609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.196539    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339aa318-ced7-44f5-929c-09fa6493e1fd-kube-api-access-fqpmj" pod "339aa318-ced7-44f5-929c-09fa6493e1fd" (UID: "339aa318-ced7-44f5-929c-09fa6493e1fd"). InnerVolumeSpecName "kube-api-access-fqpmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:35.241258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.241226    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:35.297237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.296384    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339aa318-ced7-44f5-929c-09fa6493e1fd-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:35.297237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.296419    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqpmj\" (UniqueName: \"kubernetes.io/projected/339aa318-ced7-44f5-929c-09fa6493e1fd-kube-api-access-fqpmj\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:35.884551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.884477    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"
Nov 02 01:27:35.884968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.884481    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk" event={"ID":"339aa318-ced7-44f5-929c-09fa6493e1fd","Type":"ContainerDied","Data":"13b67269bdd94dcddabe28c88aab8c8e564ee7b74447d4cc3bd2a5ecb33e1f08"}
Nov 02 01:27:35.884968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.884606    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13b67269bdd94dcddabe28c88aab8c8e564ee7b74447d4cc3bd2a5ecb33e1f08"
Nov 02 01:27:35.886206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.886182    4851 generic.go:356] "Generic (PLEG): container finished" podID="e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6" containerID="e76b5628c3c4126c7cd00d8220aa9fd806de6cc29087c3c27363b6b685aab198" exitCode=0
Nov 02 01:27:35.886356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:35.886232    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8" event={"ID":"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6","Type":"ContainerDied","Data":"e76b5628c3c4126c7cd00d8220aa9fd806de6cc29087c3c27363b6b685aab198"}
Nov 02 01:27:36.077331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:36.061490    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:36.911916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:36.911886    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:36.983769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:36.983435    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"]
Nov 02 01:27:36.984222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:36.984160    4851 cpu_manager.go:414] "containerMap: removing container" podUID="339aa318-ced7-44f5-929c-09fa6493e1fd" containerName="revoke-keys"
Nov 02 01:27:36.984222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:36.984183    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339aa318-ced7-44f5-929c-09fa6493e1fd" containerName="revoke-keys"
Nov 02 01:27:36.985057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:36.985035    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="339aa318-ced7-44f5-929c-09fa6493e1fd" containerName="revoke-keys"
Nov 02 01:27:36.988060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:36.988018    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:36.997789 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:36.997329    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"]
Nov 02 01:27:37.089364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.089340    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:37.117048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.117018    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba78bcec-t2czh\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:37.117203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.117171    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r689q\" (UniqueName: \"kubernetes.io/projected/51cc8105-e17e-4d74-979e-fb0913245396-kube-api-access-r689q\") pod \"maas-api-revoke-sub-ba78bcec-t2czh\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:37.117365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.117346    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba78bcec-t2czh\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:37.218304 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.218269    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-xks-ca\") pod \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") "
Nov 02 01:27:37.218477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.218357    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-openshift-ca\") pod \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") "
Nov 02 01:27:37.218477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.218448    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-kube-api-access-99jsm\" (UniqueName: \"kubernetes.io/projected/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-kube-api-access-99jsm\") pod \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\" (UID: \"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6\") "
Nov 02 01:27:37.218668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.218648    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba78bcec-t2czh\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:37.218761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.218705    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-openshift-ca" pod "e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6" (UID: "e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:37.218813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.218752    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r689q\" (UniqueName: \"kubernetes.io/projected/51cc8105-e17e-4d74-979e-fb0913245396-kube-api-access-r689q\") pod \"maas-api-revoke-sub-ba78bcec-t2czh\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:37.218978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.218954    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba78bcec-t2czh\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:37.219344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.219214    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:37.219439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.219369    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba78bcec-t2czh\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:37.220690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.220670    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-xks-ca" pod "e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6" (UID: "e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:37.220767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.220710    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-kube-api-access-99jsm" pod "e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6" (UID: "e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6"). InnerVolumeSpecName "kube-api-access-99jsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:37.221424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.221397    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba78bcec-t2czh\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:37.231419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.231401    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r689q\" (UniqueName: \"kubernetes.io/projected/51cc8105-e17e-4d74-979e-fb0913245396-kube-api-access-r689q\") pod \"maas-api-revoke-sub-ba78bcec-t2czh\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:37.314994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.314962    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:37.320688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.320659    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99jsm\" (UniqueName: \"kubernetes.io/projected/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-kube-api-access-99jsm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:37.320767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.320696    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:37.596541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.596515    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"]
Nov 02 01:27:37.597089 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:37.597068    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cc8105_e17e_4d74_979e_fb0913245396.slice/crio-b83930aa1a450972a83b527f1b90851b62a35aa92b095c1767707048a3e70410 WatchSource:0}: Error finding container b83930aa1a450972a83b527f1b90851b62a35aa92b095c1767707048a3e70410: Status 404 returned error can't find the container with id b83930aa1a450972a83b527f1b90851b62a35aa92b095c1767707048a3e70410
Nov 02 01:27:37.695626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.695592    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:37.900205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.900131    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"
Nov 02 01:27:37.900386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.900136    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8" event={"ID":"e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6","Type":"ContainerDied","Data":"cf0c271756eaa39ef6c31d26557e0a5d8a63f7c5a4bed7e19e345eefe4329df1"}
Nov 02 01:27:37.900386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.900253    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf0c271756eaa39ef6c31d26557e0a5d8a63f7c5a4bed7e19e345eefe4329df1"
Nov 02 01:27:37.901803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:37.901777    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh" event={"ID":"51cc8105-e17e-4d74-979e-fb0913245396","Type":"ContainerStarted","Data":"b83930aa1a450972a83b527f1b90851b62a35aa92b095c1767707048a3e70410"}
Nov 02 01:27:38.518725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:38.518670    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:38.909715 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:38.909626    4851 generic.go:356] "Generic (PLEG): container finished" podID="51cc8105-e17e-4d74-979e-fb0913245396" containerID="f72c452b60736f64524904c05720787a888699ee72adc77a70257b601574b715" exitCode=0
Nov 02 01:27:38.909878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:38.909719    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh" event={"ID":"51cc8105-e17e-4d74-979e-fb0913245396","Type":"ContainerDied","Data":"f72c452b60736f64524904c05720787a888699ee72adc77a70257b601574b715"}
Nov 02 01:27:39.453704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:39.452859    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:40.070983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.070954    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:40.115891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.115859    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:40.266091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.266012    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-xks-ca\") pod \"51cc8105-e17e-4d74-979e-fb0913245396\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") "
Nov 02 01:27:40.266267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.266102    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cc8105-e17e-4d74-979e-fb0913245396-kube-api-access-r689q\" (UniqueName: \"kubernetes.io/projected/51cc8105-e17e-4d74-979e-fb0913245396-kube-api-access-r689q\") pod \"51cc8105-e17e-4d74-979e-fb0913245396\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") "
Nov 02 01:27:40.266359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.266338    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-openshift-ca\") pod \"51cc8105-e17e-4d74-979e-fb0913245396\" (UID: \"51cc8105-e17e-4d74-979e-fb0913245396\") "
Nov 02 01:27:40.266766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.266738    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-openshift-ca" pod "51cc8105-e17e-4d74-979e-fb0913245396" (UID: "51cc8105-e17e-4d74-979e-fb0913245396"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:40.268542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.268518    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cc8105-e17e-4d74-979e-fb0913245396-kube-api-access-r689q" pod "51cc8105-e17e-4d74-979e-fb0913245396" (UID: "51cc8105-e17e-4d74-979e-fb0913245396"). InnerVolumeSpecName "kube-api-access-r689q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:40.268656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.268534    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-xks-ca" pod "51cc8105-e17e-4d74-979e-fb0913245396" (UID: "51cc8105-e17e-4d74-979e-fb0913245396"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:40.368061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.368014    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:40.368061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.368050    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r689q\" (UniqueName: \"kubernetes.io/projected/51cc8105-e17e-4d74-979e-fb0913245396-kube-api-access-r689q\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:40.368061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.368067    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cc8105-e17e-4d74-979e-fb0913245396-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:40.924749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.924714    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"
Nov 02 01:27:40.924749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.924741    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh" event={"ID":"51cc8105-e17e-4d74-979e-fb0913245396","Type":"ContainerDied","Data":"b83930aa1a450972a83b527f1b90851b62a35aa92b095c1767707048a3e70410"}
Nov 02 01:27:40.924989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.924787    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b83930aa1a450972a83b527f1b90851b62a35aa92b095c1767707048a3e70410"
Nov 02 01:27:40.967330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:40.965840    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:41.434757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.434725    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"]
Nov 02 01:27:41.435706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.435670    4851 cpu_manager.go:414] "containerMap: removing container" podUID="e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6" containerName="revoke-keys"
Nov 02 01:27:41.435779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.435706    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6" containerName="revoke-keys"
Nov 02 01:27:41.435779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.435721    4851 cpu_manager.go:414] "containerMap: removing container" podUID="51cc8105-e17e-4d74-979e-fb0913245396" containerName="revoke-keys"
Nov 02 01:27:41.435779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.435729    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cc8105-e17e-4d74-979e-fb0913245396" containerName="revoke-keys"
Nov 02 01:27:41.436246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.435881    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6" containerName="revoke-keys"
Nov 02 01:27:41.436246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.435893    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cc8105-e17e-4d74-979e-fb0913245396" containerName="revoke-keys"
Nov 02 01:27:41.440136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.440113    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.453387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.449850    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"]
Nov 02 01:27:41.453838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.453798    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5ed42b\"/\"distinct-2-w3-5ed42b-kserve-self-signed-certs\""
Nov 02 01:27:41.496889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.496857    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-tmp-dir\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.497066 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.497039    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d3a0c-8652-4445-b6e6-52d670571cfe-tls-certs\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.497107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.497093    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-dshm\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.497177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.497147    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jsq\" (UniqueName: \"kubernetes.io/projected/e54d3a0c-8652-4445-b6e6-52d670571cfe-kube-api-access-c9jsq\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.497282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.497269    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-home\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.497345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.497322    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-model-cache\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.507481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.507455    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:41.599162 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.599122    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-home\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.599391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.599194    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-model-cache\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.599391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.599253    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-tmp-dir\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.599519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.599405    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d3a0c-8652-4445-b6e6-52d670571cfe-tls-certs\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.599519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.599438    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-dshm\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.599519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.599477    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jsq\" (UniqueName: \"kubernetes.io/projected/e54d3a0c-8652-4445-b6e6-52d670571cfe-kube-api-access-c9jsq\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.600092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.600060    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-home\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.600212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.600071    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-model-cache\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.600327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.600280    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-tmp-dir\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.603274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.603241    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d3a0c-8652-4445-b6e6-52d670571cfe-tls-certs\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.603417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.603312    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-dshm\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.612420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.612391    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jsq\" (UniqueName: \"kubernetes.io/projected/e54d3a0c-8652-4445-b6e6-52d670571cfe-kube-api-access-c9jsq\") pod \"distinct-2-w3-5ed42b-kserve-785655c547-tdnsn\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") " pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.763403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.762738    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:41.996326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:41.991452    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:42.199328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:42.198827    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"]
Nov 02 01:27:42.235002 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:42.234967    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode54d3a0c_8652_4445_b6e6_52d670571cfe.slice/crio-44c676b5234e3c10450d74449c0dac57d52ffcdfddc7a9e95758a6064400368e WatchSource:0}: Error finding container 44c676b5234e3c10450d74449c0dac57d52ffcdfddc7a9e95758a6064400368e: Status 404 returned error can't find the container with id 44c676b5234e3c10450d74449c0dac57d52ffcdfddc7a9e95758a6064400368e
Nov 02 01:27:43.057311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:43.055801    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" event={"ID":"e54d3a0c-8652-4445-b6e6-52d670571cfe","Type":"ContainerStarted","Data":"85feb868483c9a8f77a74a7c6028b558a75ffeb098f7033fba35983ce5e7ca7a"}
Nov 02 01:27:43.057311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:43.055858    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" event={"ID":"e54d3a0c-8652-4445-b6e6-52d670571cfe","Type":"ContainerStarted","Data":"44c676b5234e3c10450d74449c0dac57d52ffcdfddc7a9e95758a6064400368e"}
Nov 02 01:27:43.083317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:43.079394    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podStartSLOduration=2.07937673 podStartE2EDuration="2.07937673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:43.078727288 +0000 UTC m=+768.383660116" watchObservedRunningTime="2026-11-02 01:27:43.07937673 +0000 UTC m=+768.384309558"
Nov 02 01:27:43.171446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:43.171410    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:43.763346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:43.763283    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:44.184477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.184443    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:44.319092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.319057    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"]
Nov 02 01:27:44.323489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.323457    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.328103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.328078    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:27:44.328233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.328101    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:27:44.328233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.328106    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:27:44.328233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.328199    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:27:44.332367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.332345    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"]
Nov 02 01:27:44.449841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.449759    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d40d735-bskfl\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.449995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.449868    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d40d735-bskfl\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.449995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.449917    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffhbm\" (UniqueName: \"kubernetes.io/projected/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-kube-api-access-ffhbm\") pod \"maas-api-revoke-sub-8d40d735-bskfl\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.551518 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.551481    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d40d735-bskfl\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.551690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.551570    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d40d735-bskfl\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.551690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.551608    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffhbm\" (UniqueName: \"kubernetes.io/projected/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-kube-api-access-ffhbm\") pod \"maas-api-revoke-sub-8d40d735-bskfl\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.552454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.552278    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d40d735-bskfl\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.554083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.554052    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d40d735-bskfl\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.568931 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.568907    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffhbm\" (UniqueName: \"kubernetes.io/projected/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-kube-api-access-ffhbm\") pod \"maas-api-revoke-sub-8d40d735-bskfl\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.643032 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.643003    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:44.923478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:44.923452    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"]
Nov 02 01:27:44.923660 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:44.923635    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9a1c20_8bb9_46b3_a805_db5cc2b0c44b.slice/crio-bfd309e0cc17244ad70ff45ebae79c0212094c5e0c38b70309520abc09037330 WatchSource:0}: Error finding container bfd309e0cc17244ad70ff45ebae79c0212094c5e0c38b70309520abc09037330: Status 404 returned error can't find the container with id bfd309e0cc17244ad70ff45ebae79c0212094c5e0c38b70309520abc09037330
Nov 02 01:27:45.045958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:45.045926    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:45.103752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:45.103710    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl" event={"ID":"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b","Type":"ContainerStarted","Data":"bfd309e0cc17244ad70ff45ebae79c0212094c5e0c38b70309520abc09037330"}
Nov 02 01:27:45.691772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:45.691724    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:45.843180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:45.843147    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:46.111013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:46.110929    4851 generic.go:356] "Generic (PLEG): container finished" podID="de9a1c20-8bb9-46b3-a805-db5cc2b0c44b" containerID="e3879fe43d4aef00283c427ae26bac3a72c1b97d7f59b07d3d9c8926e4eac117" exitCode=0
Nov 02 01:27:46.111142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:46.111017    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl" event={"ID":"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b","Type":"ContainerDied","Data":"e3879fe43d4aef00283c427ae26bac3a72c1b97d7f59b07d3d9c8926e4eac117"}
Nov 02 01:27:46.534921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:46.534873    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:46.720708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:46.720671    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:47.281613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.281587    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:47.324271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.324233    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-xks-ca\") pod \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") "
Nov 02 01:27:47.324463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.324358    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-kube-api-access-ffhbm\" (UniqueName: \"kubernetes.io/projected/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-kube-api-access-ffhbm\") pod \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") "
Nov 02 01:27:47.324543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.324525    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-openshift-ca\") pod \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\" (UID: \"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b\") "
Nov 02 01:27:47.324925 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.324897    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-openshift-ca" pod "de9a1c20-8bb9-46b3-a805-db5cc2b0c44b" (UID: "de9a1c20-8bb9-46b3-a805-db5cc2b0c44b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:47.326900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.326867    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-xks-ca" pod "de9a1c20-8bb9-46b3-a805-db5cc2b0c44b" (UID: "de9a1c20-8bb9-46b3-a805-db5cc2b0c44b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:47.327148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.327119    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-kube-api-access-ffhbm" pod "de9a1c20-8bb9-46b3-a805-db5cc2b0c44b" (UID: "de9a1c20-8bb9-46b3-a805-db5cc2b0c44b"). InnerVolumeSpecName "kube-api-access-ffhbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:47.426196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.426163    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:47.426196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.426197    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffhbm\" (UniqueName: \"kubernetes.io/projected/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-kube-api-access-ffhbm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:47.426433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.426212    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:47.502592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:47.502544    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:47.655074 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:27:47.655039    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:27:48.125703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:48.125672    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"
Nov 02 01:27:48.125703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:48.125700    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl" event={"ID":"de9a1c20-8bb9-46b3-a805-db5cc2b0c44b","Type":"ContainerDied","Data":"bfd309e0cc17244ad70ff45ebae79c0212094c5e0c38b70309520abc09037330"}
Nov 02 01:27:48.126203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:48.125744    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfd309e0cc17244ad70ff45ebae79c0212094c5e0c38b70309520abc09037330"
Nov 02 01:27:48.508502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:48.508469    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:49.282845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:49.279765    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:49.938593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:49.938557    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:50.641621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.641581    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:50.784024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.783989    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"]
Nov 02 01:27:50.784555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.784541    4851 cpu_manager.go:414] "containerMap: removing container" podUID="de9a1c20-8bb9-46b3-a805-db5cc2b0c44b" containerName="revoke-keys"
Nov 02 01:27:50.784612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.784557    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9a1c20-8bb9-46b3-a805-db5cc2b0c44b" containerName="revoke-keys"
Nov 02 01:27:50.784679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.784670    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9a1c20-8bb9-46b3-a805-db5cc2b0c44b" containerName="revoke-keys"
Nov 02 01:27:50.788349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.788334    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:50.793581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.793549    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:27:50.793711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.793694    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:27:50.793765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.793724    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:27:50.793815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.793763    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:27:50.797510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.797485    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"]
Nov 02 01:27:50.888805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.888772    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93fdcd4a-5hvms\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:50.889013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.888855    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93fdcd4a-5hvms\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:50.889013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.888910    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkbz\" (UniqueName: \"kubernetes.io/projected/4a763458-101d-468e-bfd3-c82efd747be1-kube-api-access-jgkbz\") pod \"maas-api-revoke-sub-93fdcd4a-5hvms\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:50.989701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.989663    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93fdcd4a-5hvms\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:50.989897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.989722    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93fdcd4a-5hvms\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:50.989897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.989742    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkbz\" (UniqueName: \"kubernetes.io/projected/4a763458-101d-468e-bfd3-c82efd747be1-kube-api-access-jgkbz\") pod \"maas-api-revoke-sub-93fdcd4a-5hvms\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:50.990456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.990436    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93fdcd4a-5hvms\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:50.992243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:50.992227    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93fdcd4a-5hvms\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:51.002102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:51.002077    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkbz\" (UniqueName: \"kubernetes.io/projected/4a763458-101d-468e-bfd3-c82efd747be1-kube-api-access-jgkbz\") pod \"maas-api-revoke-sub-93fdcd4a-5hvms\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:51.103811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:51.103779    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:51.383378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:51.383348    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"]
Nov 02 01:27:51.383724 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:51.383698    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a763458_101d_468e_bfd3_c82efd747be1.slice/crio-b55e7911618f50e542d0f38b59a949dfca2b230d0767166df5eb2ad362d178da WatchSource:0}: Error finding container b55e7911618f50e542d0f38b59a949dfca2b230d0767166df5eb2ad362d178da: Status 404 returned error can't find the container with id b55e7911618f50e542d0f38b59a949dfca2b230d0767166df5eb2ad362d178da
Nov 02 01:27:51.439429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:51.439394    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:51.763970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:51.763938    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:51.776178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:51.776150    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:52.138689 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.138656    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:52.204423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.203804    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms" event={"ID":"4a763458-101d-468e-bfd3-c82efd747be1","Type":"ContainerStarted","Data":"8fa5f3312e412de16a26dcd7dfd6080b8838c9a0476cae0dd036709e943c58aa"}
Nov 02 01:27:52.204423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.203858    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms" event={"ID":"4a763458-101d-468e-bfd3-c82efd747be1","Type":"ContainerStarted","Data":"b55e7911618f50e542d0f38b59a949dfca2b230d0767166df5eb2ad362d178da"}
Nov 02 01:27:52.215607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.215583    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:27:52.223578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.223531    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms" podStartSLOduration=2.223516261 podStartE2EDuration="2.223516261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:52.222586692 +0000 UTC m=+777.527519519" watchObservedRunningTime="2026-11-02 01:27:52.223516261 +0000 UTC m=+777.528449275"
Nov 02 01:27:52.339857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.339822    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"]
Nov 02 01:27:52.344919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.344890    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.351992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.351950    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"]
Nov 02 01:27:52.353017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.352986    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"distinct-2-w6-b9d816-kserve-self-signed-certs\""
Nov 02 01:27:52.403765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.403686    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l46z\" (UniqueName: \"kubernetes.io/projected/6781693f-2ff0-449d-b191-ec010c6cb045-kube-api-access-5l46z\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.403765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.403721    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-dshm\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.403765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.403757    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-tmp-dir\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.404001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.403887    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-home\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.404001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.403916    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6781693f-2ff0-449d-b191-ec010c6cb045-tls-certs\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.404105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.404060    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-model-cache\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.406097 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.406072    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:52.507310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.505550    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-model-cache\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.507310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.505652    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l46z\" (UniqueName: \"kubernetes.io/projected/6781693f-2ff0-449d-b191-ec010c6cb045-kube-api-access-5l46z\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.507310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.505694    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-dshm\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.507310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.505759    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-tmp-dir\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.507310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.505859    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-home\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.507310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.505888    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6781693f-2ff0-449d-b191-ec010c6cb045-tls-certs\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.507310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.507091    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-tmp-dir\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.512030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.511995    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-model-cache\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.512648 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.512617    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-home\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.525016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.524985    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-dshm\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.528267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.528233    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l46z\" (UniqueName: \"kubernetes.io/projected/6781693f-2ff0-449d-b191-ec010c6cb045-kube-api-access-5l46z\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.528903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.528809    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6781693f-2ff0-449d-b191-ec010c6cb045-tls-certs\") pod \"distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") " pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.661932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.661898    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:52.945080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.945043    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"]
Nov 02 01:27:52.948607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:52.948570    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:53.224479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:53.224361    4851 generic.go:356] "Generic (PLEG): container finished" podID="4a763458-101d-468e-bfd3-c82efd747be1" containerID="8fa5f3312e412de16a26dcd7dfd6080b8838c9a0476cae0dd036709e943c58aa" exitCode=0
Nov 02 01:27:53.224479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:53.224420    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms" event={"ID":"4a763458-101d-468e-bfd3-c82efd747be1","Type":"ContainerDied","Data":"8fa5f3312e412de16a26dcd7dfd6080b8838c9a0476cae0dd036709e943c58aa"}
Nov 02 01:27:53.227034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:53.227001    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" event={"ID":"6781693f-2ff0-449d-b191-ec010c6cb045","Type":"ContainerStarted","Data":"22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b"}
Nov 02 01:27:53.227194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:53.227047    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" event={"ID":"6781693f-2ff0-449d-b191-ec010c6cb045","Type":"ContainerStarted","Data":"c80933d4abb9fc15b0d02b5884588a26b4013501e84c9981477abb9130256e69"}
Nov 02 01:27:53.262937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:53.262872    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podStartSLOduration=1.262843449 podStartE2EDuration="1.262843449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:53.261209499 +0000 UTC m=+778.566142328" watchObservedRunningTime="2026-11-02 01:27:53.262843449 +0000 UTC m=+778.567776277"
Nov 02 01:27:53.665315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:53.663407    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:27:53.937545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:53.937460    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:54.072331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.072272    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"]
Nov 02 01:27:54.091770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.091736    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.147325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.141929    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"]
Nov 02 01:27:54.147325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.143198    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.147325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.143306    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.147325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.143340    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.147325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.143375    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.147325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.143425    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcb2a59-eee1-4301-89b7-828d25dd269d-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.147325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.143457    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdxjv\" (UniqueName: \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-kube-api-access-pdxjv\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.147325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.143488    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.147325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.143516    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.147325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.143632    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.302404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.262797    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.304323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.304278    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.304615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.304596    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.308380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.308352    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.500316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.339382    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.500316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.339992    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.500316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.340234    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.500316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.363796    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.521438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.521398    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.521759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.521736    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcb2a59-eee1-4301-89b7-828d25dd269d-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.547403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.547368    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdxjv\" (UniqueName: \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-kube-api-access-pdxjv\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.547620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.547601    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.547740 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.547725    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.552477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.522216    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.556408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.553175    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.601316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.578602    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcb2a59-eee1-4301-89b7-828d25dd269d-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.601594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.601567    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"]
Nov 02 01:27:54.613355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.609715    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdxjv\" (UniqueName: \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-kube-api-access-pdxjv\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.618392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.618357    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.625475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.621142    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"]
Nov 02 01:27:54.668189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.666961    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.668189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.667277    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.688313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.680337    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"]
Nov 02 01:27:54.688313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.682142    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"]
Nov 02 01:27:54.768333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.767409    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnlv\" (UniqueName: \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-kube-api-access-4fnlv\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.768333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.767560    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.768333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.767625    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.768333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.767707    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.768333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.767741    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f759bd-f068-415d-b56f-54c4e7a7545a-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.768333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.767766    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.768333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.767800    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768461    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768493    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768519    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768558    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768584    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768611    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768635    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768662    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf435d77-26a8-436f-b3fc-b0a1235bd325-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768709    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768740    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8lk\" (UniqueName: \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-kube-api-access-ws8lk\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.768879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.768776    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.916826    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.916882    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f759bd-f068-415d-b56f-54c4e7a7545a-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.916907    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.916939    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.916976    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.916997    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917023    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917054    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917078    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917102    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917125    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917150    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf435d77-26a8-436f-b3fc-b0a1235bd325-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917193    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917221    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8lk\" (UniqueName: \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-kube-api-access-ws8lk\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917255    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917304    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnlv\" (UniqueName: \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-kube-api-access-4fnlv\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917451    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917468    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917495    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.917994    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.922069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.918251    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.926871 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.926828    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.927229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.927207    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.927414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.927228    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf435d77-26a8-436f-b3fc-b0a1235bd325-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.927873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.927849    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:54.928050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.927887    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.928869 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.928836    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:54.929009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:54.928943    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:55.015685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.015560    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:55.027777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.026831    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnlv\" (UniqueName: \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-kube-api-access-4fnlv\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:55.030919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.030894    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:55.110362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.099686    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f759bd-f068-415d-b56f-54c4e7a7545a-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:55.110362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.108865    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:55.205847 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.204919    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:55.206035 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.206011    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8lk\" (UniqueName: \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-kube-api-access-ws8lk\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:55.209021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.208995    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:55.211691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.211659    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:55.256915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.215835    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:55.271271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.271239    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:55.329145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.329023    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:55.361746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.353222    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:55.446800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.446749    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-openshift-ca\") pod \"4a763458-101d-468e-bfd3-c82efd747be1\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") "
Nov 02 01:27:55.447173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.447151    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a763458-101d-468e-bfd3-c82efd747be1-kube-api-access-jgkbz\" (UniqueName: \"kubernetes.io/projected/4a763458-101d-468e-bfd3-c82efd747be1-kube-api-access-jgkbz\") pod \"4a763458-101d-468e-bfd3-c82efd747be1\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") "
Nov 02 01:27:55.447553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.447516    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-xks-ca\") pod \"4a763458-101d-468e-bfd3-c82efd747be1\" (UID: \"4a763458-101d-468e-bfd3-c82efd747be1\") "
Nov 02 01:27:55.449118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.449095    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-openshift-ca" pod "4a763458-101d-468e-bfd3-c82efd747be1" (UID: "4a763458-101d-468e-bfd3-c82efd747be1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:55.464329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.464275    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a763458-101d-468e-bfd3-c82efd747be1-kube-api-access-jgkbz" pod "4a763458-101d-468e-bfd3-c82efd747be1" (UID: "4a763458-101d-468e-bfd3-c82efd747be1"). InnerVolumeSpecName "kube-api-access-jgkbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:55.481901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.481867    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-xks-ca" pod "4a763458-101d-468e-bfd3-c82efd747be1" (UID: "4a763458-101d-468e-bfd3-c82efd747be1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:55.516019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.515732    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:55.564319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.561983    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:55.564319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.562022    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgkbz\" (UniqueName: \"kubernetes.io/projected/4a763458-101d-468e-bfd3-c82efd747be1-kube-api-access-jgkbz\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:55.564319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.562038    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a763458-101d-468e-bfd3-c82efd747be1-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:27:55.647331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.646634    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms" event={"ID":"4a763458-101d-468e-bfd3-c82efd747be1","Type":"ContainerDied","Data":"b55e7911618f50e542d0f38b59a949dfca2b230d0767166df5eb2ad362d178da"}
Nov 02 01:27:55.647331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.646740    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b55e7911618f50e542d0f38b59a949dfca2b230d0767166df5eb2ad362d178da"
Nov 02 01:27:55.647331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.646979    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"
Nov 02 01:27:55.847557 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:55.847500    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcb2a59_eee1_4301_89b7_828d25dd269d.slice/crio-b021349c17514e6b351a71463507e8fefa5d647a262404063eacbe97492b73f5 WatchSource:0}: Error finding container b021349c17514e6b351a71463507e8fefa5d647a262404063eacbe97492b73f5: Status 404 returned error can't find the container with id b021349c17514e6b351a71463507e8fefa5d647a262404063eacbe97492b73f5
Nov 02 01:27:55.847833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.847812    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"]
Nov 02 01:27:55.872152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.870689    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:27:55.872152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.870775    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:27:55.872152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.870817    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:27:55.872152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.871197    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:27:55.872152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.871266    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:27:55.872152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.871344    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:27:55.873447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.873422    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"]
Nov 02 01:27:55.903642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.903615    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"]
Nov 02 01:27:55.912607 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:27:55.912581    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f759bd_f068_415d_b56f_54c4e7a7545a.slice/crio-5a35740a98b91dbdad0d8aff49bf8f1eabc0044d464297d7e1b5fa46ec5d3bca WatchSource:0}: Error finding container 5a35740a98b91dbdad0d8aff49bf8f1eabc0044d464297d7e1b5fa46ec5d3bca: Status 404 returned error can't find the container with id 5a35740a98b91dbdad0d8aff49bf8f1eabc0044d464297d7e1b5fa46ec5d3bca
Nov 02 01:27:55.916587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.916545    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:27:55.916699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.916640    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:27:55.916699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:55.916686    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:27:56.330776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.330740    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:56.655435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.655323    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" event={"ID":"ffcb2a59-eee1-4301-89b7-828d25dd269d","Type":"ContainerStarted","Data":"fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68"}
Nov 02 01:27:56.655602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.655436    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" event={"ID":"ffcb2a59-eee1-4301-89b7-828d25dd269d","Type":"ContainerStarted","Data":"b021349c17514e6b351a71463507e8fefa5d647a262404063eacbe97492b73f5"}
Nov 02 01:27:56.657263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.657234    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" event={"ID":"cf435d77-26a8-436f-b3fc-b0a1235bd325","Type":"ContainerStarted","Data":"9c27755317ff9ab84e50a4404e328bcb0704761a71fde910c9975e7f7c81e582"}
Nov 02 01:27:56.657448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.657280    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" event={"ID":"cf435d77-26a8-436f-b3fc-b0a1235bd325","Type":"ContainerStarted","Data":"cf3ada1ee33b7d75b5a16a405493f70bc1bfee4206a1a3aa55734a2223261f71"}
Nov 02 01:27:56.658981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.658946    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" event={"ID":"a3f759bd-f068-415d-b56f-54c4e7a7545a","Type":"ContainerStarted","Data":"ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb"}
Nov 02 01:27:56.659107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.659009    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" event={"ID":"a3f759bd-f068-415d-b56f-54c4e7a7545a","Type":"ContainerStarted","Data":"5a35740a98b91dbdad0d8aff49bf8f1eabc0044d464297d7e1b5fa46ec5d3bca"}
Nov 02 01:27:56.718828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.714941    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" podStartSLOduration=2.7149215460000002 podStartE2EDuration="2.714921546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:56.71327193 +0000 UTC m=+782.018204753" watchObservedRunningTime="2026-11-02 01:27:56.714921546 +0000 UTC m=+782.019854370"
Nov 02 01:27:56.718828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.715407    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" podStartSLOduration=2.715391502 podStartE2EDuration="2.715391502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:56.688346823 +0000 UTC m=+781.993279649" watchObservedRunningTime="2026-11-02 01:27:56.715391502 +0000 UTC m=+782.020324329"
Nov 02 01:27:56.926506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.926474    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:27:56.927770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.927747    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:27:56.927880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:56.927823    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 01:27:57.187535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:57.187335    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:57.293723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:57.293687    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:27:57.312425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:57.312391    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:27:57.312688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:57.312658    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 01:27:57.338323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:57.338002    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:27:57.342316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:57.339160    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 01:27:57.342316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:57.339212    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 01:27:57.926835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:57.926794    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:27:57.927034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:57.926860    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 01:27:58.224405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:58.224230    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:58.273162 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:58.273124    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:27:58.273532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:58.273489    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 01:27:58.331171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:58.331140    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 01:27:58.331825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:58.331519    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 01:27:58.926054 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:58.926025    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:27:58.926534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:58.926509    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 01:27:59.255826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:59.255732    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:27:59.272751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:59.272718    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:27:59.273028 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:59.273000    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 01:27:59.337209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:59.337173    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 01:27:59.337596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:59.337562    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 01:27:59.926336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:59.926282    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:27:59.926790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:27:59.926768    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 01:28:00.275373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:00.274685    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:28:00.275373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:00.274750    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 01:28:00.329850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:00.329816    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 01:28:00.330087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:00.330065    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 01:28:00.509465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:00.509430    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:00.926768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:00.926451    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:28:00.926768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:00.926498    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 01:28:01.276527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:01.276011    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:28:01.276527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:01.276072    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 01:28:01.330487 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:01.330452    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 01:28:01.330653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:01.330504    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 01:28:01.601370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:01.600399    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:01.926526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:01.926495    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:28:01.926775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:01.926547    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 01:28:02.273367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.272786    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 01:28:02.273367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.272863    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 01:28:02.330987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.330942    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 01:28:02.331175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.331003    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 01:28:02.634063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.633932    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:02.665132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.665105    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:28:02.764216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.764177    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:28:02.813865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.813813    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" podStartSLOduration=8.813797229 podStartE2EDuration="8.813797229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:56.751738267 +0000 UTC m=+782.056671093" watchObservedRunningTime="2026-11-02 01:28:02.813797229 +0000 UTC m=+788.118730055"
Nov 02 01:28:02.832945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.832917    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:28:02.927164 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.926033    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 02 01:28:02.927164 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:02.926103    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 02 01:28:03.467538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:03.466271    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:28:03.467538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:03.467399    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:28:03.469464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:03.469442    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:28:03.530714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:03.530663    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:03.806862 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:03.806747    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:28:03.807804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:03.807781    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:28:03.809084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:03.809066    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:28:03.930910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:03.930882    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:28:03.931336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:03.931317    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:28:03.932150 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:03.932129    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:28:04.330751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:04.330705    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:05.025216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:05.025183    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:05.471066 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:05.471008    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:05.937710 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:05.937678    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:06.707450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:06.707417    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:07.510644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:07.510610    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:08.258239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.258204    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:08.857077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.857049    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"]
Nov 02 01:28:08.857609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.857593    4851 cpu_manager.go:414] "containerMap: removing container" podUID="4a763458-101d-468e-bfd3-c82efd747be1" containerName="revoke-keys"
Nov 02 01:28:08.857668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.857611    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a763458-101d-468e-bfd3-c82efd747be1" containerName="revoke-keys"
Nov 02 01:28:08.857719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.857710    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a763458-101d-468e-bfd3-c82efd747be1" containerName="revoke-keys"
Nov 02 01:28:08.860913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.860897    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:08.875675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.875648    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"]
Nov 02 01:28:08.926660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.926620    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:08.926850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.926746    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:08.926850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.926781    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:08.926934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.926906    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:08.927055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.927032    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:08.927143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.927128    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:08.927181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.927169    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kj4d\" (UniqueName: \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-kube-api-access-8kj4d\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:08.927324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.927265    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:08.927411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:08.927359    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.028219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028186    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.028461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028237    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kj4d\" (UniqueName: \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-kube-api-access-8kj4d\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.028461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028274    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.028461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028312    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.028461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028339    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.028461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028384    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.028737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028516    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.028737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028606    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.028737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028674    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.028876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028734    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.029018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.028993    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.029360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.029037    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.029360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.029279    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.029508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.029410    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.031108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.031086    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.031108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.031102    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.040471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.040446    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.040577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.040567    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kj4d\" (UniqueName: \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-kube-api-access-8kj4d\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.099548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.099517    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:09.177648 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.177615    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:09.470112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.470085    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"]
Nov 02 01:28:09.471252 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:09.471218    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256d8fe9_92f6_42ed_9a7b_e7f80d55ae2b.slice/crio-0e21ab406f96b196565e68799226cf45ce2390745f3be89669b3e5d23bbe5699 WatchSource:0}: Error finding container 0e21ab406f96b196565e68799226cf45ce2390745f3be89669b3e5d23bbe5699: Status 404 returned error can't find the container with id 0e21ab406f96b196565e68799226cf45ce2390745f3be89669b3e5d23bbe5699
Nov 02 01:28:09.473830 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.473797    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:09.473920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.473863    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:09.473920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.473893    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:09.864605 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.864506    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" event={"ID":"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b","Type":"ContainerStarted","Data":"cd0b18a5667aad25db9afd9b677900e4430d2363921dde4c082e42b5ff77d468"}
Nov 02 01:28:09.864605 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.864565    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" event={"ID":"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b","Type":"ContainerStarted","Data":"0e21ab406f96b196565e68799226cf45ce2390745f3be89669b3e5d23bbe5699"}
Nov 02 01:28:09.890211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.890158    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podStartSLOduration=1.890143846 podStartE2EDuration="1.890143846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:09.887772806 +0000 UTC m=+795.192705632" watchObservedRunningTime="2026-11-02 01:28:09.890143846 +0000 UTC m=+795.195076673"
Nov 02 01:28:09.903824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:09.903794    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:10.178375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:10.178337    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:10.180009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:10.179985    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:28:10.180132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:10.180032    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 01:28:10.720764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:10.720732    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:11.178659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.178616    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:28:11.178873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.178687    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 01:28:11.475948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.475864    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:11.884619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.884537    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"]
Nov 02 01:28:11.889385 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.889361    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:11.895238 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.895208    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47dd19\"/\"distinct-2-w2-47dd19-kserve-self-signed-certs\""
Nov 02 01:28:11.899588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.899558    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"]
Nov 02 01:28:11.961048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.960961    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:11.988366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.988270    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-model-cache\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:11.988656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.988524    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-dshm\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:11.988733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.988709    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf185485-9f8b-4745-8c17-bd099588f5d4-tls-certs\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:11.988782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.988768    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lr72\" (UniqueName: \"kubernetes.io/projected/bf185485-9f8b-4745-8c17-bd099588f5d4-kube-api-access-4lr72\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:11.988852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.988836    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-home\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:11.988928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:11.988913    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-tmp-dir\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.095330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.094023    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf185485-9f8b-4745-8c17-bd099588f5d4-tls-certs\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.095330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.094090    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lr72\" (UniqueName: \"kubernetes.io/projected/bf185485-9f8b-4745-8c17-bd099588f5d4-kube-api-access-4lr72\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.095330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.094125    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-home\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.095330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.094190    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-tmp-dir\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.095330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.094278    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-model-cache\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.095330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.094323    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-dshm\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.095330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.094760    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-home\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.095966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.095537    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-model-cache\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.099385 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.096089    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-tmp-dir\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.116350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.112081    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-dshm\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.162441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.146487    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lr72\" (UniqueName: \"kubernetes.io/projected/bf185485-9f8b-4745-8c17-bd099588f5d4-kube-api-access-4lr72\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.185147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.185111    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:28:12.185817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.185415    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 01:28:12.186805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.186535    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf185485-9f8b-4745-8c17-bd099588f5d4-tls-certs\") pod \"distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.205695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.205660    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:12.319092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.319059    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:12.520188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.520154    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"]
Nov 02 01:28:12.520711 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:12.520684    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf185485_9f8b_4745_8c17_bd099588f5d4.slice/crio-838c93390cdf044e8403aa87a057d7a78a1d9687ac1caf82794f67e8431d28dd WatchSource:0}: Error finding container 838c93390cdf044e8403aa87a057d7a78a1d9687ac1caf82794f67e8431d28dd: Status 404 returned error can't find the container with id 838c93390cdf044e8403aa87a057d7a78a1d9687ac1caf82794f67e8431d28dd
Nov 02 01:28:12.890921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.890820    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" event={"ID":"bf185485-9f8b-4745-8c17-bd099588f5d4","Type":"ContainerStarted","Data":"9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443"}
Nov 02 01:28:12.890921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.890873    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" event={"ID":"bf185485-9f8b-4745-8c17-bd099588f5d4","Type":"ContainerStarted","Data":"838c93390cdf044e8403aa87a057d7a78a1d9687ac1caf82794f67e8431d28dd"}
Nov 02 01:28:12.912142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:12.912089    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podStartSLOduration=1.912076216 podStartE2EDuration="1.912076216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:12.909748507 +0000 UTC m=+798.214681333" watchObservedRunningTime="2026-11-02 01:28:12.912076216 +0000 UTC m=+798.217009079"
Nov 02 01:28:13.178914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:13.178872    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:28:13.179133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:13.178937    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 01:28:13.207353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:13.207319    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:13.216100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:13.216071    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:13.896485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:13.896444    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:14.158070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:14.153722    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:14.178906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:14.178874    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:28:14.179073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:14.178931    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 01:28:14.813358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:14.813326    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:15.185323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:15.184758    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:28:15.185323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:15.184817    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 01:28:15.613646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:15.613551    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:16.183726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:16.183689    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:28:16.198334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:16.187363    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 01:28:16.989316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:16.989053    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:17.822315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:17.821805    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:17.925317 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:28:17.917520    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:28:18.111406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:18.107804    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:18.189319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:18.188400    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:28:18.189319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:18.188481    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:18.311072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:18.311025    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:28:18.414382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:18.405387    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:18.437850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:18.437791    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:18.899549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:18.899428    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:19.229489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:19.229449    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:28:19.229688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:19.229521    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:19.380489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:19.380457    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:19.380877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:19.380859    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:19.381568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:19.381549    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:28:19.631556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:19.631446    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:20.341069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:20.341035    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:21.130440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.130403    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:21.480325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.480263    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"]
Nov 02 01:28:21.485950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.485926    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.495283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.495252    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"]
Nov 02 01:28:21.496897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.496872    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ca1c53\"/\"distinct-2-w0-ca1c53-kserve-self-signed-certs\""
Nov 02 01:28:21.556507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.556472    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:21.584508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.584473    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8wh\" (UniqueName: \"kubernetes.io/projected/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-kube-api-access-dc8wh\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.584666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.584525    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tls-certs\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.584666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.584620    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tmp-dir\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.584802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.584669    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-model-cache\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.584802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.584718    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-dshm\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.584802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.584794    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-home\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.686170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.686124    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8wh\" (UniqueName: \"kubernetes.io/projected/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-kube-api-access-dc8wh\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.686928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.686901    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tls-certs\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.687142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.687121    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tmp-dir\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.687331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.687284    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-model-cache\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.687460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.687447    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-dshm\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.687927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.687765    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-model-cache\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.688039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.687826    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tmp-dir\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.688184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.688168    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-home\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.692496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.689005    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-home\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.698745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.698716    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tls-certs\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.714127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.711095    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8wh\" (UniqueName: \"kubernetes.io/projected/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-kube-api-access-dc8wh\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.734042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.732038    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-dshm\") pod \"distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.804244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.804209    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:21.964240 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:21.964202    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:22.087060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.087029    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"]
Nov 02 01:28:22.087283 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:22.087262    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00fd0bde_f5d1_4a50_ad48_85bc6881cb27.slice/crio-895523fc9b4a2bab0005b7c3e37c15a15f31a8f333609e62fee6377c9b6a4ab3 WatchSource:0}: Error finding container 895523fc9b4a2bab0005b7c3e37c15a15f31a8f333609e62fee6377c9b6a4ab3: Status 404 returned error can't find the container with id 895523fc9b4a2bab0005b7c3e37c15a15f31a8f333609e62fee6377c9b6a4ab3
Nov 02 01:28:22.198904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.198860    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" event={"ID":"00fd0bde-f5d1-4a50-ad48-85bc6881cb27","Type":"ContainerStarted","Data":"945de5f2a496a130b017e9a25ef8d6fe584256d53162bc0c6c9c18c5817faf15"}
Nov 02 01:28:22.199087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.198914    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" event={"ID":"00fd0bde-f5d1-4a50-ad48-85bc6881cb27","Type":"ContainerStarted","Data":"895523fc9b4a2bab0005b7c3e37c15a15f31a8f333609e62fee6377c9b6a4ab3"}
Nov 02 01:28:22.200875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.200850    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"]
Nov 02 01:28:22.205515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.205495    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.206559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.206540    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:22.211703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.211679    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:28:22.211833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.211792    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:28:22.211900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.211846    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:28:22.211900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.211882    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:28:22.217071 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.217048    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"]
Nov 02 01:28:22.224536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.224485    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podStartSLOduration=1.224467416 podStartE2EDuration="1.224467416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:22.219459076 +0000 UTC m=+807.524391903" watchObservedRunningTime="2026-11-02 01:28:22.224467416 +0000 UTC m=+807.529400243"
Nov 02 01:28:22.230148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.230113    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:22.302735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.302699    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c040c3d5-qqpwz\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.302928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.302747    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c040c3d5-qqpwz\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.302928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.302871    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6kj\" (UniqueName: \"kubernetes.io/projected/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-kube-api-access-2s6kj\") pod \"maas-api-revoke-sub-c040c3d5-qqpwz\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.404189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.404077    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c040c3d5-qqpwz\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.404189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.404144    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c040c3d5-qqpwz\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.404515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.404213    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6kj\" (UniqueName: \"kubernetes.io/projected/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-kube-api-access-2s6kj\") pod \"maas-api-revoke-sub-c040c3d5-qqpwz\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.409795 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.405999    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c040c3d5-qqpwz\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.412755 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.411742    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c040c3d5-qqpwz\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.420040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.420007    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6kj\" (UniqueName: \"kubernetes.io/projected/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-kube-api-access-2s6kj\") pod \"maas-api-revoke-sub-c040c3d5-qqpwz\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.532707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.532668    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:22.753181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.752993    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:22.811258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.811220    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:22.838163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:22.838121    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"]
Nov 02 01:28:22.838509 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:22.838485    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47d4cb3_e2b8_40cb_8ba6_73c8bc1baacb.slice/crio-c672f2e73484f7eef4e0a77502588d149bd5e2f3262e0a12201243bd0c357f20 WatchSource:0}: Error finding container c672f2e73484f7eef4e0a77502588d149bd5e2f3262e0a12201243bd0c357f20: Status 404 returned error can't find the container with id c672f2e73484f7eef4e0a77502588d149bd5e2f3262e0a12201243bd0c357f20
Nov 02 01:28:23.209493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:23.209455    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz" event={"ID":"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb","Type":"ContainerStarted","Data":"51037fc6f21730927d0a292d964120352aa2a39ebbe1bd1bdc884cceef7475e7"}
Nov 02 01:28:23.209677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:23.209531    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz" event={"ID":"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb","Type":"ContainerStarted","Data":"c672f2e73484f7eef4e0a77502588d149bd5e2f3262e0a12201243bd0c357f20"}
Nov 02 01:28:23.224185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:23.224159    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:28:23.251644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:23.251584    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz" podStartSLOduration=1.251564999 podStartE2EDuration="1.251564999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:23.22946404 +0000 UTC m=+808.534396868" watchObservedRunningTime="2026-11-02 01:28:23.251564999 +0000 UTC m=+808.556497834"
Nov 02 01:28:23.617618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:23.617582    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:24.219056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:24.219022    4851 generic.go:356] "Generic (PLEG): container finished" podID="c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb" containerID="51037fc6f21730927d0a292d964120352aa2a39ebbe1bd1bdc884cceef7475e7" exitCode=0
Nov 02 01:28:24.219278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:24.219107    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz" event={"ID":"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb","Type":"ContainerDied","Data":"51037fc6f21730927d0a292d964120352aa2a39ebbe1bd1bdc884cceef7475e7"}
Nov 02 01:28:24.338711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:24.338677    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:25.035538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.035505    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:25.436077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.436046    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:25.492333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.492284    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-xks-ca\") pod \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") "
Nov 02 01:28:25.492531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.492390    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-openshift-ca\") pod \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") "
Nov 02 01:28:25.492593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.492567    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-kube-api-access-2s6kj\" (UniqueName: \"kubernetes.io/projected/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-kube-api-access-2s6kj\") pod \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\" (UID: \"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb\") "
Nov 02 01:28:25.492872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.492841    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-openshift-ca" pod "c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb" (UID: "c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:28:25.494761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.494731    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-xks-ca" pod "c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb" (UID: "c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:28:25.495250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.495227    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-kube-api-access-2s6kj" pod "c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb" (UID: "c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb"). InnerVolumeSpecName "kube-api-access-2s6kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:28:25.594403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.594267    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s6kj\" (UniqueName: \"kubernetes.io/projected/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-kube-api-access-2s6kj\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:25.594403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.594349    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:25.594403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.594360    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:25.825353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:25.825321    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:26.099553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:26.099506    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:26.263574 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:26.260004    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz" event={"ID":"c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb","Type":"ContainerDied","Data":"c672f2e73484f7eef4e0a77502588d149bd5e2f3262e0a12201243bd0c357f20"}
Nov 02 01:28:26.263574 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:26.260100    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c672f2e73484f7eef4e0a77502588d149bd5e2f3262e0a12201243bd0c357f20"
Nov 02 01:28:26.263574 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:26.260204    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"
Nov 02 01:28:26.647616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:26.647581    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:27.530170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:27.530127    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:28.222632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:28.222584    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:29.117785 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:29.117733    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:29.952672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:29.952625    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:30.719956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:30.719928    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:31.510738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:31.510710    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:31.804882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:31.804805    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:31.817414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:31.817388    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:32.202815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.202783    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:32.347559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.347534    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:28:32.681402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.681363    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"]
Nov 02 01:28:32.682076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.682056    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb" containerName="revoke-keys"
Nov 02 01:28:32.682162 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.682078    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb" containerName="revoke-keys"
Nov 02 01:28:32.682266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.682254    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb" containerName="revoke-keys"
Nov 02 01:28:32.686799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.686777    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.694279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.694165    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-123986\"/\"openshift-service-ca.crt\""
Nov 02 01:28:32.695613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.695263    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-123986\"/\"distinct-2-w5-123986-kserve-self-signed-certs\""
Nov 02 01:28:32.695613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.695492    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-123986\"/\"default-dockercfg-fhs95\""
Nov 02 01:28:32.695776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.695756    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-123986\"/\"kube-root-ca.crt\""
Nov 02 01:28:32.696160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.696140    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"]
Nov 02 01:28:32.751408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.751374    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:32.799645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.799611    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-dshm\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.799838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.799668    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-home\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.799838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.799814    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-model-cache\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.799921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.799844    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tmp-dir\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.799921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.799859    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tls-certs\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.800007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.799992    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9h4\" (UniqueName: \"kubernetes.io/projected/0af7a9a6-d01b-47cc-9762-0db8296f0b54-kube-api-access-5w9h4\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.912321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.908577    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9h4\" (UniqueName: \"kubernetes.io/projected/0af7a9a6-d01b-47cc-9762-0db8296f0b54-kube-api-access-5w9h4\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.912321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.909126    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-dshm\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.912321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.909876    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-home\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.912321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.910031    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-home\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.912321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.910781    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-model-cache\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.912321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.910862    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-model-cache\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.912321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.910974    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tmp-dir\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.912321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.911271    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tls-certs\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.921682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.921648    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tmp-dir\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.937049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.936955    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-dshm\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.942211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.942179    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9h4\" (UniqueName: \"kubernetes.io/projected/0af7a9a6-d01b-47cc-9762-0db8296f0b54-kube-api-access-5w9h4\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:32.954360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:32.951505    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tls-certs\") pod \"distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") " pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:33.001625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.001591    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:33.132998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.132964    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:33.286416 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:33.286384    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0af7a9a6_d01b_47cc_9762_0db8296f0b54.slice/crio-9e596dc00c814f8a9afc47e1da0c063ece957368cacc27ea0499e33aed70c2f5 WatchSource:0}: Error finding container 9e596dc00c814f8a9afc47e1da0c063ece957368cacc27ea0499e33aed70c2f5: Status 404 returned error can't find the container with id 9e596dc00c814f8a9afc47e1da0c063ece957368cacc27ea0499e33aed70c2f5
Nov 02 01:28:33.286615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.286598    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"]
Nov 02 01:28:33.349237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.349213    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" event={"ID":"0af7a9a6-d01b-47cc-9762-0db8296f0b54","Type":"ContainerStarted","Data":"9e596dc00c814f8a9afc47e1da0c063ece957368cacc27ea0499e33aed70c2f5"}
Nov 02 01:28:33.710987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.710950    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"]
Nov 02 01:28:33.715095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.715077    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:33.719338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.719314    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:28:33.724332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.720479    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:28:33.724332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.720527    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:28:33.724332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.720620    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:28:33.725281 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.725257    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"]
Nov 02 01:28:33.839473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.839443    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d86131d1-km25q\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:33.839648 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.839596    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg95h\" (UniqueName: \"kubernetes.io/projected/3b994aa4-ded5-4b3b-b825-dcb433421abe-kube-api-access-tg95h\") pod \"maas-api-revoke-sub-d86131d1-km25q\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:33.839692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.839652    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d86131d1-km25q\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:33.891182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.891143    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:33.944318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.941364    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg95h\" (UniqueName: \"kubernetes.io/projected/3b994aa4-ded5-4b3b-b825-dcb433421abe-kube-api-access-tg95h\") pod \"maas-api-revoke-sub-d86131d1-km25q\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:33.944318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.941475    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d86131d1-km25q\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:33.944318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.941616    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d86131d1-km25q\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:33.944318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.943844    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d86131d1-km25q\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:33.971969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.962226    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d86131d1-km25q\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:33.982354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:33.982229    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg95h\" (UniqueName: \"kubernetes.io/projected/3b994aa4-ded5-4b3b-b825-dcb433421abe-kube-api-access-tg95h\") pod \"maas-api-revoke-sub-d86131d1-km25q\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:34.030567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:34.030533    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:34.311425 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:34.311390    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b994aa4_ded5_4b3b_b825_dcb433421abe.slice/crio-07eeb9a97607aa1acebed1481093a950aaf4cce868a6ef4ad2f67a4a224c5c9a WatchSource:0}: Error finding container 07eeb9a97607aa1acebed1481093a950aaf4cce868a6ef4ad2f67a4a224c5c9a: Status 404 returned error can't find the container with id 07eeb9a97607aa1acebed1481093a950aaf4cce868a6ef4ad2f67a4a224c5c9a
Nov 02 01:28:34.311425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:34.311410    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"]
Nov 02 01:28:34.358312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:34.358266    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q" event={"ID":"3b994aa4-ded5-4b3b-b825-dcb433421abe","Type":"ContainerStarted","Data":"07eeb9a97607aa1acebed1481093a950aaf4cce868a6ef4ad2f67a4a224c5c9a"}
Nov 02 01:28:34.359813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:34.359792    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" event={"ID":"0af7a9a6-d01b-47cc-9762-0db8296f0b54","Type":"ContainerStarted","Data":"6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba"}
Nov 02 01:28:34.381625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:34.381580    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podStartSLOduration=2.381569364 podStartE2EDuration="2.381569364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:34.37827144 +0000 UTC m=+819.683204267" watchObservedRunningTime="2026-11-02 01:28:34.381569364 +0000 UTC m=+819.686502190"
Nov 02 01:28:34.643264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:34.643235    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:35.002177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:35.002148    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:35.329696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:35.329602    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:35.419007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:35.417376    4851 generic.go:356] "Generic (PLEG): container finished" podID="3b994aa4-ded5-4b3b-b825-dcb433421abe" containerID="1affcd6ba679a12640de37d645b7702b13f0884aa4b5431ddb9bd9c9703470c2" exitCode=0
Nov 02 01:28:35.419007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:35.418905    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q" event={"ID":"3b994aa4-ded5-4b3b-b825-dcb433421abe","Type":"ContainerDied","Data":"1affcd6ba679a12640de37d645b7702b13f0884aa4b5431ddb9bd9c9703470c2"}
Nov 02 01:28:36.231327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:36.228790    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:37.061330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.059551    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:37.139356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.124885    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-xks-ca\") pod \"3b994aa4-ded5-4b3b-b825-dcb433421abe\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") "
Nov 02 01:28:37.139356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.125027    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-openshift-ca\") pod \"3b994aa4-ded5-4b3b-b825-dcb433421abe\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") "
Nov 02 01:28:37.139356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.135260    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-openshift-ca" pod "3b994aa4-ded5-4b3b-b825-dcb433421abe" (UID: "3b994aa4-ded5-4b3b-b825-dcb433421abe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:28:37.139356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.135606    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b994aa4-ded5-4b3b-b825-dcb433421abe-kube-api-access-tg95h\" (UniqueName: \"kubernetes.io/projected/3b994aa4-ded5-4b3b-b825-dcb433421abe-kube-api-access-tg95h\") pod \"3b994aa4-ded5-4b3b-b825-dcb433421abe\" (UID: \"3b994aa4-ded5-4b3b-b825-dcb433421abe\") "
Nov 02 01:28:37.139356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.136262    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:37.171321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.170525    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b994aa4-ded5-4b3b-b825-dcb433421abe-kube-api-access-tg95h" pod "3b994aa4-ded5-4b3b-b825-dcb433421abe" (UID: "3b994aa4-ded5-4b3b-b825-dcb433421abe"). InnerVolumeSpecName "kube-api-access-tg95h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:28:37.199335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.190105    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-xks-ca" pod "3b994aa4-ded5-4b3b-b825-dcb433421abe" (UID: "3b994aa4-ded5-4b3b-b825-dcb433421abe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:28:37.238319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.237502    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b994aa4-ded5-4b3b-b825-dcb433421abe-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:37.238319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.237537    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg95h\" (UniqueName: \"kubernetes.io/projected/3b994aa4-ded5-4b3b-b825-dcb433421abe-kube-api-access-tg95h\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:37.264469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.256741    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:37.603863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.603828    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q" event={"ID":"3b994aa4-ded5-4b3b-b825-dcb433421abe","Type":"ContainerDied","Data":"07eeb9a97607aa1acebed1481093a950aaf4cce868a6ef4ad2f67a4a224c5c9a"}
Nov 02 01:28:37.604146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.604125    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07eeb9a97607aa1acebed1481093a950aaf4cce868a6ef4ad2f67a4a224c5c9a"
Nov 02 01:28:37.604357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:37.604342    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"
Nov 02 01:28:38.226784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:38.225741    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:38.248185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:38.245576    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:28:38.248185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:38.247533    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:28:38.366407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:38.366362    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:38.390033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:38.389445    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:38.424311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:38.424241    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:38.475321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:38.432412    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:28:38.502734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:38.502546    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:38.556022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:38.554968    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:28:39.161531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.161480    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"]
Nov 02 01:28:39.162734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.162703    4851 cpu_manager.go:414] "containerMap: removing container" podUID="3b994aa4-ded5-4b3b-b825-dcb433421abe" containerName="revoke-keys"
Nov 02 01:28:39.162734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.162733    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b994aa4-ded5-4b3b-b825-dcb433421abe" containerName="revoke-keys"
Nov 02 01:28:39.162959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.162902    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b994aa4-ded5-4b3b-b825-dcb433421abe" containerName="revoke-keys"
Nov 02 01:28:39.174542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.174513    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.181966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.181932    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"]
Nov 02 01:28:39.331314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.330689    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:39.386321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.379701    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.386321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.379756    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.386321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.379814    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.386321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.379889    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.386321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.379918    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58p9\" (UniqueName: \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-kube-api-access-q58p9\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.386321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.379995    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.386321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.380032    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.386321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.380077    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.386321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.380168    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.409936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.406551    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"]
Nov 02 01:28:39.417785 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.417026    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.457313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.456548    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"]
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.487388    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.487438    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58p9\" (UniqueName: \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-kube-api-access-q58p9\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.487514    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.487548    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.487592    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.487667    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.487739    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.487765    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.487819    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.488264    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.489143    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.489710    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.489845    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.496361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.490341    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.503081 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.503015    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58p9\" (UniqueName: \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-kube-api-access-q58p9\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.504408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.504360    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.509871 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.509841    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.529181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.521596    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:39.589315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.589101    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc860c6b-3659-4fce-b6ab-401e1264fc67-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.589315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.589206    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.589315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.589241    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.593312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.590381    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.593312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.590419    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvbx\" (UniqueName: \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-kube-api-access-5cvbx\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.593312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.590501    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.593312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.590542    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.593312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.590565    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.593312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.590598    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.695368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.695071    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.695368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.695139    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.695368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.695167    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.695368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.695197    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.695368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.695330    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc860c6b-3659-4fce-b6ab-401e1264fc67-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.695734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.695412    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.695734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.695443    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.695734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.695556    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.695734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.695582    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvbx\" (UniqueName: \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-kube-api-access-5cvbx\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.698316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.696388    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.698316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.696635    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.715395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.715362    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvbx\" (UniqueName: \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-kube-api-access-5cvbx\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.715875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.715851    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.724320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.722596    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.727324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.725424    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc860c6b-3659-4fce-b6ab-401e1264fc67-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.755312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.744805    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.755312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.753616    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.778313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.769216    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:39.818313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:39.818049    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:40.063575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.063476    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:40.109754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.109718    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"]
Nov 02 01:28:40.122786 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.122725    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.123371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.123347    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"]
Nov 02 01:28:40.141334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.141233    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:28:40.143999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.141577    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:28:40.143999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.143132    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:28:40.143999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.143352    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:28:40.159121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.159087    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:40.159235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.159156    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:40.159235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.159186    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:40.161683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.161664    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"]
Nov 02 01:28:40.223821 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.223792    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1529a9a5-75mq5\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.224006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.223829    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1529a9a5-75mq5\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.224006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.223918    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7pj\" (UniqueName: \"kubernetes.io/projected/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-kube-api-access-zt7pj\") pod \"maas-api-revoke-sub-1529a9a5-75mq5\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.325007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.324915    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1529a9a5-75mq5\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.325007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.324965    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1529a9a5-75mq5\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.325237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.325100    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7pj\" (UniqueName: \"kubernetes.io/projected/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-kube-api-access-zt7pj\") pod \"maas-api-revoke-sub-1529a9a5-75mq5\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.325826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.325798    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1529a9a5-75mq5\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.327611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.327588    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1529a9a5-75mq5\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.336411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.336373    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:40.338179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.338154    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7pj\" (UniqueName: \"kubernetes.io/projected/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-kube-api-access-zt7pj\") pod \"maas-api-revoke-sub-1529a9a5-75mq5\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.422458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.419056    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"]
Nov 02 01:28:40.429381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.429212    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:40.431618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.431591    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:40.431818 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.431798    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:40.464093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.464068    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:40.764752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.764716    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" event={"ID":"bc860c6b-3659-4fce-b6ab-401e1264fc67","Type":"ContainerStarted","Data":"a8f8284f26f5e5cf2a3a6c68a558b360b4cc5baff4d458bbe6edcf1842927494"}
Nov 02 01:28:40.764952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.764778    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" event={"ID":"bc860c6b-3659-4fce-b6ab-401e1264fc67","Type":"ContainerStarted","Data":"180319cf1a8435acabb7022ed5054b530410c194c99b1f8ee319c55cb0d1156d"}
Nov 02 01:28:40.766920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.766893    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" event={"ID":"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524","Type":"ContainerStarted","Data":"763f1dc8bf4d638ff7e266004c26036551a8cda3bf4a54bbcf547b0a0dca0ffa"}
Nov 02 01:28:40.767083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.766943    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" event={"ID":"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524","Type":"ContainerStarted","Data":"aa53bd191f8c881c636cd0c836566f58a77bae0f03668510d6db2f4f59b6c13f"}
Nov 02 01:28:40.821896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.821867    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:40.828605 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.828581    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"]
Nov 02 01:28:40.829134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.829096    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podStartSLOduration=1.829083105 podStartE2EDuration="1.829083105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:40.817151811 +0000 UTC m=+826.122084664" watchObservedRunningTime="2026-11-02 01:28:40.829083105 +0000 UTC m=+826.134015930"
Nov 02 01:28:40.882225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:40.882161    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" podStartSLOduration=1.882143134 podStartE2EDuration="1.882143134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:40.876831679 +0000 UTC m=+826.181764507" watchObservedRunningTime="2026-11-02 01:28:40.882143134 +0000 UTC m=+826.187075961"
Nov 02 01:28:41.066651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:41.066567    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:41.068415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:41.068387    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:28:41.068561 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:41.068434    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:28:41.506269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:41.506213    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:41.794488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:41.794395    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5" event={"ID":"8352c9c4-9b10-4dd7-b632-970bc5e9c46f","Type":"ContainerStarted","Data":"a8b4033335cd2b045df811d69b41657d2a34563a9d39c47626cf67351ec09e76"}
Nov 02 01:28:41.794488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:41.794450    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5" event={"ID":"8352c9c4-9b10-4dd7-b632-970bc5e9c46f","Type":"ContainerStarted","Data":"384ed87b76ac821e795bcda4652954b02723f15251c0f74a26d5f744b6768c48"}
Nov 02 01:28:41.815130 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:41.815077    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5" podStartSLOduration=1.815062336 podStartE2EDuration="1.815062336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:41.812512297 +0000 UTC m=+827.117445130" watchObservedRunningTime="2026-11-02 01:28:41.815062336 +0000 UTC m=+827.119995222"
Nov 02 01:28:41.821342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:41.821310    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 01:28:41.821558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:41.821537    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 01:28:42.086228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:42.085761    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:28:42.086228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:42.085826    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:28:42.363361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:42.363202    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:42.801019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:42.800982    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:42.805070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:42.805041    4851 generic.go:356] "Generic (PLEG): container finished" podID="8352c9c4-9b10-4dd7-b632-970bc5e9c46f" containerID="a8b4033335cd2b045df811d69b41657d2a34563a9d39c47626cf67351ec09e76" exitCode=0
Nov 02 01:28:42.805224 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:42.805129    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5" event={"ID":"8352c9c4-9b10-4dd7-b632-970bc5e9c46f","Type":"ContainerDied","Data":"a8b4033335cd2b045df811d69b41657d2a34563a9d39c47626cf67351ec09e76"}
Nov 02 01:28:42.819839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:42.819802    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 01:28:42.820007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:42.819873    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 01:28:43.011863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:43.011830    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:43.072730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:43.068729    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:28:43.072730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:43.068787    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:28:43.191782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:43.191751    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:43.711947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:43.711903    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:43.823323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:43.821803    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 01:28:43.823323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:43.821861    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 01:28:43.906454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:43.906424    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:28:44.065315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.064677    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:28:44.065315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.064740    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:28:44.293917 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.293857    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:44.308208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.308180    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-kube-api-access-zt7pj\" (UniqueName: \"kubernetes.io/projected/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-kube-api-access-zt7pj\") pod \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") "
Nov 02 01:28:44.308362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.308257    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-xks-ca\") pod \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") "
Nov 02 01:28:44.308433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.308420    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-openshift-ca\") pod \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\" (UID: \"8352c9c4-9b10-4dd7-b632-970bc5e9c46f\") "
Nov 02 01:28:44.308774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.308756    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-openshift-ca" pod "8352c9c4-9b10-4dd7-b632-970bc5e9c46f" (UID: "8352c9c4-9b10-4dd7-b632-970bc5e9c46f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:28:44.310596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.310574    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-kube-api-access-zt7pj" pod "8352c9c4-9b10-4dd7-b632-970bc5e9c46f" (UID: "8352c9c4-9b10-4dd7-b632-970bc5e9c46f"). InnerVolumeSpecName "kube-api-access-zt7pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:28:44.310959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.310933    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-xks-ca" pod "8352c9c4-9b10-4dd7-b632-970bc5e9c46f" (UID: "8352c9c4-9b10-4dd7-b632-970bc5e9c46f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:28:44.409870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.409841    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:44.409870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.409866    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:44.409870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.409879    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt7pj\" (UniqueName: \"kubernetes.io/projected/8352c9c4-9b10-4dd7-b632-970bc5e9c46f-kube-api-access-zt7pj\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:44.819749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.819704    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 01:28:44.819945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.819766    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 01:28:44.914374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.911089    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"
Nov 02 01:28:44.914374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.911119    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5" event={"ID":"8352c9c4-9b10-4dd7-b632-970bc5e9c46f","Type":"ContainerDied","Data":"384ed87b76ac821e795bcda4652954b02723f15251c0f74a26d5f744b6768c48"}
Nov 02 01:28:44.914374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.911165    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="384ed87b76ac821e795bcda4652954b02723f15251c0f74a26d5f744b6768c48"
Nov 02 01:28:44.990548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:44.990453    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:45.065314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:45.064999    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:28:45.065314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:45.065090    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:28:45.790989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:45.790950    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:45.819944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:45.819913    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 01:28:45.820116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:45.819974    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 01:28:45.984316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:45.976891    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:46.071366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:46.070760    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:28:46.071366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:46.070821    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:28:46.750317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:46.748313    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:46.824211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:46.824121    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 01:28:46.824531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:46.824501    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 01:28:47.064765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:47.064677    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 01:28:47.065143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:47.064754    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 01:28:47.559885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:47.559853    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:47.722687 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:28:47.722651    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:28:48.188978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:48.188937    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:48.189544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:48.189526    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:48.190571 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:48.190551    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:28:48.358937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:48.358911    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:48.359537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:48.359475    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:48.360254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:48.360236    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:28:48.427330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:48.427282    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:49.217432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:49.217400    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:50.009087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:50.009052    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:50.725257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:50.725222    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:51.432201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:51.432160    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:52.121767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:52.121736    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:52.926320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:52.926251    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:53.760003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:53.759953    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:53.940887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:53.940852    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"]
Nov 02 01:28:53.941754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:53.941730    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8352c9c4-9b10-4dd7-b632-970bc5e9c46f" containerName="revoke-keys"
Nov 02 01:28:53.941754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:53.941754    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8352c9c4-9b10-4dd7-b632-970bc5e9c46f" containerName="revoke-keys"
Nov 02 01:28:53.941975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:53.941961    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8352c9c4-9b10-4dd7-b632-970bc5e9c46f" containerName="revoke-keys"
Nov 02 01:28:53.946893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:53.946867    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:53.959214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:53.959182    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"]
Nov 02 01:28:54.037747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.037660    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.037747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.037720    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmwxp\" (UniqueName: \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-kube-api-access-bmwxp\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.037954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.037871    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.037954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.037897    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac1c433-8412-47ae-9794-ce298c3593bd-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.038072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.037963    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac1c433-8412-47ae-9794-ce298c3593bd-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.038072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.038059    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.038238 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.038118    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.038238 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.038183    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.038238 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.038211    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.139502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.139469    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.139656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.139513    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmwxp\" (UniqueName: \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-kube-api-access-bmwxp\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.139656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.139561    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.139656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.139575    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac1c433-8412-47ae-9794-ce298c3593bd-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.139656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.139631    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac1c433-8412-47ae-9794-ce298c3593bd-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.139852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.139670    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.139852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.139711    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.139852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.139766    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.139852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.139795    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.140059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.139983    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.140146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.140124    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.140198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.140175    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.140380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.140356    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.140454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.140373    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac1c433-8412-47ae-9794-ce298c3593bd-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.142325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.142263    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.142611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.142592    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac1c433-8412-47ae-9794-ce298c3593bd-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.154364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.154338    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.154565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.154547    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmwxp\" (UniqueName: \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-kube-api-access-bmwxp\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.276302 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.276254    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:54.414426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.414395    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"]
Nov 02 01:28:54.419789 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.419761    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.427301 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.427260    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"]
Nov 02 01:28:54.428878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.428587    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:28:54.428878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.428591    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:28:54.428878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.428596    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:28:54.428878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.428627    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:28:54.444052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.444023    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d3d1d0-c2rsh\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.444231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.444213    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz94l\" (UniqueName: \"kubernetes.io/projected/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-kube-api-access-qz94l\") pod \"maas-api-revoke-sub-77d3d1d0-c2rsh\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.444365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.444347    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d3d1d0-c2rsh\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.465922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.465888    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"]
Nov 02 01:28:54.471323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.471278    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.485210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.485176    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"]
Nov 02 01:28:54.545396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545322    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.545396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545363    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.545578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545458    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d3d1d0-c2rsh\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.545578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545515    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.545578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545569    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.545683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545658    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.545723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545712    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz94l\" (UniqueName: \"kubernetes.io/projected/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-kube-api-access-qz94l\") pod \"maas-api-revoke-sub-77d3d1d0-c2rsh\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.545805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545784    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.545843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545817    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.545971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545953    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d3d1d0-c2rsh\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.546046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.545990    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.546046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.546033    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqkx\" (UniqueName: \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-kube-api-access-9fqkx\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.546598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.546576    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d3d1d0-c2rsh\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.547956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.547939    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d3d1d0-c2rsh\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.564168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.564142    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz94l\" (UniqueName: \"kubernetes.io/projected/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-kube-api-access-qz94l\") pod \"maas-api-revoke-sub-77d3d1d0-c2rsh\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.564594 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:54.564565    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac1c433_8412_47ae_9794_ce298c3593bd.slice/crio-cddbfa25d3d589c2832cf95a23ad4f79a6273206eff9b41918f0f06a73807d25 WatchSource:0}: Error finding container cddbfa25d3d589c2832cf95a23ad4f79a6273206eff9b41918f0f06a73807d25: Status 404 returned error can't find the container with id cddbfa25d3d589c2832cf95a23ad4f79a6273206eff9b41918f0f06a73807d25
Nov 02 01:28:54.565223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.565188    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"]
Nov 02 01:28:54.567199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.567165    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:54.567265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.567229    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:54.567265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.567256    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:54.621173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.621145    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:54.651317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.650337    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.651317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.650422    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.651317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.650482    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.651317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.650561    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.651317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.650589    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.651317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.650687    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.651317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.650737    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqkx\" (UniqueName: \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-kube-api-access-9fqkx\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.651317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.650801    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.651317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.650834    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.655228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.655169    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.655855 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.655830    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.656095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.655900    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.656708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.656675    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.660332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.660284    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.661942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.661912    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.662554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.662534    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.690316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.675529    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.708314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.701064    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqkx\" (UniqueName: \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-kube-api-access-9fqkx\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:54.741278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.741243    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:54.795811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:54.795532    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:55.002451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.002416    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" event={"ID":"dac1c433-8412-47ae-9794-ce298c3593bd","Type":"ContainerStarted","Data":"e0afa3651a8dd6ee13b0fdcf2f7f45566f8306f7a34cc81cf039b83a75db127b"}
Nov 02 01:28:55.002636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.002474    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" event={"ID":"dac1c433-8412-47ae-9794-ce298c3593bd","Type":"ContainerStarted","Data":"cddbfa25d3d589c2832cf95a23ad4f79a6273206eff9b41918f0f06a73807d25"}
Nov 02 01:28:55.028393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.028324    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" podStartSLOduration=2.02830565 podStartE2EDuration="2.02830565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:55.026505218 +0000 UTC m=+840.331438045" watchObservedRunningTime="2026-11-02 01:28:55.02830565 +0000 UTC m=+840.333238469"
Nov 02 01:28:55.052428 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:55.052398    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9bb6b2_4311_4380_baaa_1f42d64a4aeb.slice/crio-f1d85b8e6bceaff5150533d3c8fa90dfd3997b4cde44a04b68bcb07bec2c48af WatchSource:0}: Error finding container f1d85b8e6bceaff5150533d3c8fa90dfd3997b4cde44a04b68bcb07bec2c48af: Status 404 returned error can't find the container with id f1d85b8e6bceaff5150533d3c8fa90dfd3997b4cde44a04b68bcb07bec2c48af
Nov 02 01:28:55.052844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.052821    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"]
Nov 02 01:28:55.121239 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:55.121197    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e02f029_1d4f_42ca_bca9_b963ce25eee3.slice/crio-c39da7a3ffdec8d8775ef58cb6b0334f241dcd55a0e34b7f80a1b294e77a59f2 WatchSource:0}: Error finding container c39da7a3ffdec8d8775ef58cb6b0334f241dcd55a0e34b7f80a1b294e77a59f2: Status 404 returned error can't find the container with id c39da7a3ffdec8d8775ef58cb6b0334f241dcd55a0e34b7f80a1b294e77a59f2
Nov 02 01:28:55.123440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.122916    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"]
Nov 02 01:28:55.123959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.123927    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:55.124046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.123993    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:55.124046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.124038    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:28:55.276742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.276718    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:28:55.278123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.278091    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:28:55.278241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.278139    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 01:28:55.656365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.647143    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:55.867254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.867219    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"]
Nov 02 01:28:55.871719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.871701    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:55.875193 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.875163    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzr55\" (UniqueName: \"kubernetes.io/projected/0d88eb42-dce9-4c57-adf8-a548eec1bedd-kube-api-access-bzr55\") pod \"maas-api-revoke-sub-c3cd804f-rf6jb\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:55.875381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.875357    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3cd804f-rf6jb\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:55.875808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.875645    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3cd804f-rf6jb\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:55.881031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.881010    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"]
Nov 02 01:28:55.976596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.976558    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3cd804f-rf6jb\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:55.976763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.976619    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzr55\" (UniqueName: \"kubernetes.io/projected/0d88eb42-dce9-4c57-adf8-a548eec1bedd-kube-api-access-bzr55\") pod \"maas-api-revoke-sub-c3cd804f-rf6jb\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:55.976763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.976701    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3cd804f-rf6jb\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:55.977322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.977281    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3cd804f-rf6jb\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:55.979223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.979203    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3cd804f-rf6jb\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:55.989558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:55.989531    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzr55\" (UniqueName: \"kubernetes.io/projected/0d88eb42-dce9-4c57-adf8-a548eec1bedd-kube-api-access-bzr55\") pod \"maas-api-revoke-sub-c3cd804f-rf6jb\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:56.011838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.011803    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" event={"ID":"4e02f029-1d4f-42ca-bca9-b963ce25eee3","Type":"ContainerStarted","Data":"8c84a22d516eb203c97ff0791d2743487f5249a6679deee1c6fcf44651a8547c"}
Nov 02 01:28:56.011981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.011862    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" event={"ID":"4e02f029-1d4f-42ca-bca9-b963ce25eee3","Type":"ContainerStarted","Data":"c39da7a3ffdec8d8775ef58cb6b0334f241dcd55a0e34b7f80a1b294e77a59f2"}
Nov 02 01:28:56.013623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.013598    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh" event={"ID":"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb","Type":"ContainerStarted","Data":"bd5d4a9471dc78f9dfac5e7c6816db0451bcb45d76d272762273c306d48b4ba1"}
Nov 02 01:28:56.013736 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.013636    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh" event={"ID":"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb","Type":"ContainerStarted","Data":"f1d85b8e6bceaff5150533d3c8fa90dfd3997b4cde44a04b68bcb07bec2c48af"}
Nov 02 01:28:56.065197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.065146    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" podStartSLOduration=2.065132516 podStartE2EDuration="2.065132516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:56.060519786 +0000 UTC m=+841.365452612" watchObservedRunningTime="2026-11-02 01:28:56.065132516 +0000 UTC m=+841.370065341"
Nov 02 01:28:56.079163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.078728    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh" podStartSLOduration=2.078711451 podStartE2EDuration="2.078711451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:56.077608743 +0000 UTC m=+841.382541570" watchObservedRunningTime="2026-11-02 01:28:56.078711451 +0000 UTC m=+841.383644279"
Nov 02 01:28:56.195094 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.195052    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:56.289124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.288995    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:28:56.289124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.289053    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 01:28:56.587858 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:56.586921    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d88eb42_dce9_4c57_adf8_a548eec1bedd.slice/crio-3ea7699665037de84996fef4e6332703defd3039dfa8f8a6997fbf1b466b1711 WatchSource:0}: Error finding container 3ea7699665037de84996fef4e6332703defd3039dfa8f8a6997fbf1b466b1711: Status 404 returned error can't find the container with id 3ea7699665037de84996fef4e6332703defd3039dfa8f8a6997fbf1b466b1711
Nov 02 01:28:56.593490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.593467    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"]
Nov 02 01:28:56.797697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.797661    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:28:56.799047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.799012    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:28:56.799177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:56.799086    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 01:28:57.023052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:57.023015    4851 generic.go:356] "Generic (PLEG): container finished" podID="bd9bb6b2-4311-4380-baaa-1f42d64a4aeb" containerID="bd5d4a9471dc78f9dfac5e7c6816db0451bcb45d76d272762273c306d48b4ba1" exitCode=0
Nov 02 01:28:57.023576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:57.023059    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh" event={"ID":"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb","Type":"ContainerDied","Data":"bd5d4a9471dc78f9dfac5e7c6816db0451bcb45d76d272762273c306d48b4ba1"}
Nov 02 01:28:57.024655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:57.024634    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb" event={"ID":"0d88eb42-dce9-4c57-adf8-a548eec1bedd","Type":"ContainerStarted","Data":"3ea7699665037de84996fef4e6332703defd3039dfa8f8a6997fbf1b466b1711"}
Nov 02 01:28:57.068331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:57.060283    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:57.285579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:57.285127    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:28:57.285579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:57.285185    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 01:28:57.801505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:57.800937    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:28:57.801505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:57.801037    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 01:28:58.050331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.050206    4851 generic.go:356] "Generic (PLEG): container finished" podID="0d88eb42-dce9-4c57-adf8-a548eec1bedd" containerID="0f1f9eeed062df6b11ae7de9a4a2dca1e0ffb46f2ad38528a6d13df5f7d1dd68" exitCode=0
Nov 02 01:28:58.057397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.053510    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb" event={"ID":"0d88eb42-dce9-4c57-adf8-a548eec1bedd","Type":"ContainerDied","Data":"0f1f9eeed062df6b11ae7de9a4a2dca1e0ffb46f2ad38528a6d13df5f7d1dd68"}
Nov 02 01:28:58.280889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.280851    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:28:58.281085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.280920    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 01:28:58.409828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.408480    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:58.511900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.505386    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:28:58.712190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.712146    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-kube-api-access-qz94l\" (UniqueName: \"kubernetes.io/projected/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-kube-api-access-qz94l\") pod \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") "
Nov 02 01:28:58.712387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.712224    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-xks-ca\") pod \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") "
Nov 02 01:28:58.712387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.712244    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-openshift-ca\") pod \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\" (UID: \"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb\") "
Nov 02 01:28:58.713135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.713111    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-openshift-ca" pod "bd9bb6b2-4311-4380-baaa-1f42d64a4aeb" (UID: "bd9bb6b2-4311-4380-baaa-1f42d64a4aeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:28:58.716472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.716430    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-xks-ca" pod "bd9bb6b2-4311-4380-baaa-1f42d64a4aeb" (UID: "bd9bb6b2-4311-4380-baaa-1f42d64a4aeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:28:58.716592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.716540    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-kube-api-access-qz94l" pod "bd9bb6b2-4311-4380-baaa-1f42d64a4aeb" (UID: "bd9bb6b2-4311-4380-baaa-1f42d64a4aeb"). InnerVolumeSpecName "kube-api-access-qz94l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:28:58.716592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.716553    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"]
Nov 02 01:28:58.717570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.717548    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bd9bb6b2-4311-4380-baaa-1f42d64a4aeb" containerName="revoke-keys"
Nov 02 01:28:58.717654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.717571    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9bb6b2-4311-4380-baaa-1f42d64a4aeb" containerName="revoke-keys"
Nov 02 01:28:58.717772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.717758    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9bb6b2-4311-4380-baaa-1f42d64a4aeb" containerName="revoke-keys"
Nov 02 01:28:58.722238 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.722216    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:58.722568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.722545    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"]
Nov 02 01:28:58.796999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.796941    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:28:58.797108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.796994    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 01:28:58.814991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.814962    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9319d3bd-55nw4\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:58.815197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.815183    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9319d3bd-55nw4\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:58.815248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.815236    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5z67\" (UniqueName: \"kubernetes.io/projected/3119f15b-88ea-41ab-a4fa-c4a128051041-kube-api-access-q5z67\") pod \"maas-api-revoke-sub-9319d3bd-55nw4\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:58.815528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.815514    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz94l\" (UniqueName: \"kubernetes.io/projected/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-kube-api-access-qz94l\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:58.815586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.815530    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:58.815586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.815539    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:58.917752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.917722    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9319d3bd-55nw4\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:58.917911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.917785    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5z67\" (UniqueName: \"kubernetes.io/projected/3119f15b-88ea-41ab-a4fa-c4a128051041-kube-api-access-q5z67\") pod \"maas-api-revoke-sub-9319d3bd-55nw4\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:58.917911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.917877    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9319d3bd-55nw4\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:58.918701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.918676    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9319d3bd-55nw4\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:58.920908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.920887    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9319d3bd-55nw4\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:58.929796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:58.929775    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5z67\" (UniqueName: \"kubernetes.io/projected/3119f15b-88ea-41ab-a4fa-c4a128051041-kube-api-access-q5z67\") pod \"maas-api-revoke-sub-9319d3bd-55nw4\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:59.089373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.088383    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh" event={"ID":"bd9bb6b2-4311-4380-baaa-1f42d64a4aeb","Type":"ContainerDied","Data":"f1d85b8e6bceaff5150533d3c8fa90dfd3997b4cde44a04b68bcb07bec2c48af"}
Nov 02 01:28:59.089373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.088449    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"
Nov 02 01:28:59.089373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.088457    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1d85b8e6bceaff5150533d3c8fa90dfd3997b4cde44a04b68bcb07bec2c48af"
Nov 02 01:28:59.106726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.106701    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:28:59.291510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.290563    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:28:59.291510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.290620    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 01:28:59.524133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.524101    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:28:59.629788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.629710    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-openshift-ca\") pod \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") "
Nov 02 01:28:59.630468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.630438    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d88eb42-dce9-4c57-adf8-a548eec1bedd-kube-api-access-bzr55\" (UniqueName: \"kubernetes.io/projected/0d88eb42-dce9-4c57-adf8-a548eec1bedd-kube-api-access-bzr55\") pod \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") "
Nov 02 01:28:59.630630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.630512    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-xks-ca\") pod \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\" (UID: \"0d88eb42-dce9-4c57-adf8-a548eec1bedd\") "
Nov 02 01:28:59.631475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.631451    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-openshift-ca" pod "0d88eb42-dce9-4c57-adf8-a548eec1bedd" (UID: "0d88eb42-dce9-4c57-adf8-a548eec1bedd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:28:59.690568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.690495    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d88eb42-dce9-4c57-adf8-a548eec1bedd-kube-api-access-bzr55" pod "0d88eb42-dce9-4c57-adf8-a548eec1bedd" (UID: "0d88eb42-dce9-4c57-adf8-a548eec1bedd"). InnerVolumeSpecName "kube-api-access-bzr55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:28:59.690754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.690582    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-xks-ca" pod "0d88eb42-dce9-4c57-adf8-a548eec1bedd" (UID: "0d88eb42-dce9-4c57-adf8-a548eec1bedd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:28:59.791319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.790986    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"]
Nov 02 01:28:59.791522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.791346    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:59.791522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.791370    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88eb42-dce9-4c57-adf8-a548eec1bedd-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:59.791522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.791388    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzr55\" (UniqueName: \"kubernetes.io/projected/0d88eb42-dce9-4c57-adf8-a548eec1bedd-kube-api-access-bzr55\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:28:59.796121 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:28:59.796091    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3119f15b_88ea_41ab_a4fa_c4a128051041.slice/crio-c3132cd8258440394c3493ecb5a854cc12116e2fcf42a0851e8058d4ecd97851 WatchSource:0}: Error finding container c3132cd8258440394c3493ecb5a854cc12116e2fcf42a0851e8058d4ecd97851: Status 404 returned error can't find the container with id c3132cd8258440394c3493ecb5a854cc12116e2fcf42a0851e8058d4ecd97851
Nov 02 01:28:59.796749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.796724    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:28:59.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:28:59.796775    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 01:29:00.066937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:00.066890    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:00.174513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:00.166328    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4" event={"ID":"3119f15b-88ea-41ab-a4fa-c4a128051041","Type":"ContainerStarted","Data":"c3132cd8258440394c3493ecb5a854cc12116e2fcf42a0851e8058d4ecd97851"}
Nov 02 01:29:00.174513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:00.168600    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb" event={"ID":"0d88eb42-dce9-4c57-adf8-a548eec1bedd","Type":"ContainerDied","Data":"3ea7699665037de84996fef4e6332703defd3039dfa8f8a6997fbf1b466b1711"}
Nov 02 01:29:00.174513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:00.168645    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ea7699665037de84996fef4e6332703defd3039dfa8f8a6997fbf1b466b1711"
Nov 02 01:29:00.174513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:00.168707    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"
Nov 02 01:29:00.282658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:00.282173    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:29:00.282658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:00.282232    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 01:29:00.796712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:00.796675    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:29:00.796934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:00.796756    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 01:29:01.180898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.180861    4851 generic.go:356] "Generic (PLEG): container finished" podID="3119f15b-88ea-41ab-a4fa-c4a128051041" containerID="48e14f9e0a78cc09e1271bbb6ee21102d44108d9d48e5a6ddc2bf54fbc32e7a6" exitCode=0
Nov 02 01:29:01.181408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.180952    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4" event={"ID":"3119f15b-88ea-41ab-a4fa-c4a128051041","Type":"ContainerDied","Data":"48e14f9e0a78cc09e1271bbb6ee21102d44108d9d48e5a6ddc2bf54fbc32e7a6"}
Nov 02 01:29:01.276645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.276615    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:29:01.276825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.276665    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 01:29:01.366036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.358347    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"]
Nov 02 01:29:01.366036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.359310    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0d88eb42-dce9-4c57-adf8-a548eec1bedd" containerName="revoke-keys"
Nov 02 01:29:01.366036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.359329    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d88eb42-dce9-4c57-adf8-a548eec1bedd" containerName="revoke-keys"
Nov 02 01:29:01.366036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.359539    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d88eb42-dce9-4c57-adf8-a548eec1bedd" containerName="revoke-keys"
Nov 02 01:29:01.373351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.371452    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"]
Nov 02 01:29:01.373351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.371553    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.427839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.427801    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7llt\" (UniqueName: \"kubernetes.io/projected/f6c35bed-145e-4b2c-88b2-0c5963d2df88-kube-api-access-z7llt\") pod \"maas-api-revoke-sub-79741054-twmnq\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.427982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.427901    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79741054-twmnq\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.428088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.428070    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79741054-twmnq\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.530108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.530072    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7llt\" (UniqueName: \"kubernetes.io/projected/f6c35bed-145e-4b2c-88b2-0c5963d2df88-kube-api-access-z7llt\") pod \"maas-api-revoke-sub-79741054-twmnq\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.530315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.530173    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79741054-twmnq\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.530418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.530399    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79741054-twmnq\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.530965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.530942    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79741054-twmnq\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.541319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.533173    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79741054-twmnq\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.563360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.555281    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7llt\" (UniqueName: \"kubernetes.io/projected/f6c35bed-145e-4b2c-88b2-0c5963d2df88-kube-api-access-z7llt\") pod \"maas-api-revoke-sub-79741054-twmnq\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.687989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.687953    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:01.797318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.796779    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 02 01:29:01.797318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.796836    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 02 01:29:01.976243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:01.976216    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"]
Nov 02 01:29:01.976800 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:01.976769    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c35bed_145e_4b2c_88b2_0c5963d2df88.slice/crio-187cd1a817a00701b89ca28814c8facd24041387d884fbbebd1fa42b08bbd2fd WatchSource:0}: Error finding container 187cd1a817a00701b89ca28814c8facd24041387d884fbbebd1fa42b08bbd2fd: Status 404 returned error can't find the container with id 187cd1a817a00701b89ca28814c8facd24041387d884fbbebd1fa42b08bbd2fd
Nov 02 01:29:02.030769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.030739    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:02.217129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.217088    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq" event={"ID":"f6c35bed-145e-4b2c-88b2-0c5963d2df88","Type":"ContainerStarted","Data":"187cd1a817a00701b89ca28814c8facd24041387d884fbbebd1fa42b08bbd2fd"}
Nov 02 01:29:02.493111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.493073    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:29:02.576568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.576458    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-xks-ca\") pod \"3119f15b-88ea-41ab-a4fa-c4a128051041\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") "
Nov 02 01:29:02.576750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.576584    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3119f15b-88ea-41ab-a4fa-c4a128051041-kube-api-access-q5z67\" (UniqueName: \"kubernetes.io/projected/3119f15b-88ea-41ab-a4fa-c4a128051041-kube-api-access-q5z67\") pod \"3119f15b-88ea-41ab-a4fa-c4a128051041\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") "
Nov 02 01:29:02.576863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.576834    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-openshift-ca\") pod \"3119f15b-88ea-41ab-a4fa-c4a128051041\" (UID: \"3119f15b-88ea-41ab-a4fa-c4a128051041\") "
Nov 02 01:29:02.577802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.577759    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-openshift-ca" pod "3119f15b-88ea-41ab-a4fa-c4a128051041" (UID: "3119f15b-88ea-41ab-a4fa-c4a128051041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:02.580943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.580904    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-xks-ca" pod "3119f15b-88ea-41ab-a4fa-c4a128051041" (UID: "3119f15b-88ea-41ab-a4fa-c4a128051041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:02.593744 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.593703    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3119f15b-88ea-41ab-a4fa-c4a128051041-kube-api-access-q5z67" pod "3119f15b-88ea-41ab-a4fa-c4a128051041" (UID: "3119f15b-88ea-41ab-a4fa-c4a128051041"). InnerVolumeSpecName "kube-api-access-q5z67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:02.679320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.679255    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:02.679497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.679343    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5z67\" (UniqueName: \"kubernetes.io/projected/3119f15b-88ea-41ab-a4fa-c4a128051041-kube-api-access-q5z67\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:02.679497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.679364    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3119f15b-88ea-41ab-a4fa-c4a128051041-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:02.713186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.713109    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:29:02.714489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.714466    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:29:02.715223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:02.715199    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:29:03.227620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:03.227587    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"
Nov 02 01:29:03.227620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:03.227603    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4" event={"ID":"3119f15b-88ea-41ab-a4fa-c4a128051041","Type":"ContainerDied","Data":"c3132cd8258440394c3493ecb5a854cc12116e2fcf42a0851e8058d4ecd97851"}
Nov 02 01:29:03.228159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:03.227657    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3132cd8258440394c3493ecb5a854cc12116e2fcf42a0851e8058d4ecd97851"
Nov 02 01:29:03.229895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:03.229870    4851 generic.go:356] "Generic (PLEG): container finished" podID="f6c35bed-145e-4b2c-88b2-0c5963d2df88" containerID="7b09c184dea8f11ce1fc4fd5697202ca2d31e0196b05b9a31475c39b8ed988e2" exitCode=0
Nov 02 01:29:03.230040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:03.229960    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq" event={"ID":"f6c35bed-145e-4b2c-88b2-0c5963d2df88","Type":"ContainerDied","Data":"7b09c184dea8f11ce1fc4fd5697202ca2d31e0196b05b9a31475c39b8ed988e2"}
Nov 02 01:29:03.242179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:03.242157    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:29:03.242560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:03.242516    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:29:03.243536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:03.243506    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:29:03.493581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:03.493503    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:04.433368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:04.433343    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:04.526698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:04.526619    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c35bed-145e-4b2c-88b2-0c5963d2df88-kube-api-access-z7llt\" (UniqueName: \"kubernetes.io/projected/f6c35bed-145e-4b2c-88b2-0c5963d2df88-kube-api-access-z7llt\") pod \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") "
Nov 02 01:29:04.526875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:04.526711    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-openshift-ca\") pod \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") "
Nov 02 01:29:04.527040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:04.527021    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-xks-ca\") pod \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\" (UID: \"f6c35bed-145e-4b2c-88b2-0c5963d2df88\") "
Nov 02 01:29:04.527119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:04.527065    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-openshift-ca" pod "f6c35bed-145e-4b2c-88b2-0c5963d2df88" (UID: "f6c35bed-145e-4b2c-88b2-0c5963d2df88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:04.527565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:04.527541    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:04.529023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:04.529001    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c35bed-145e-4b2c-88b2-0c5963d2df88-kube-api-access-z7llt" pod "f6c35bed-145e-4b2c-88b2-0c5963d2df88" (UID: "f6c35bed-145e-4b2c-88b2-0c5963d2df88"). InnerVolumeSpecName "kube-api-access-z7llt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:04.529278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:04.529261    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-xks-ca" pod "f6c35bed-145e-4b2c-88b2-0c5963d2df88" (UID: "f6c35bed-145e-4b2c-88b2-0c5963d2df88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:04.628803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:04.628767    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7llt\" (UniqueName: \"kubernetes.io/projected/f6c35bed-145e-4b2c-88b2-0c5963d2df88-kube-api-access-z7llt\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:04.628803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:04.628807    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c35bed-145e-4b2c-88b2-0c5963d2df88-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:05.023512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:05.023413    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:05.295362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:05.285920    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"
Nov 02 01:29:05.295362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:05.286044    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq" event={"ID":"f6c35bed-145e-4b2c-88b2-0c5963d2df88","Type":"ContainerDied","Data":"187cd1a817a00701b89ca28814c8facd24041387d884fbbebd1fa42b08bbd2fd"}
Nov 02 01:29:05.295362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:05.286119    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="187cd1a817a00701b89ca28814c8facd24041387d884fbbebd1fa42b08bbd2fd"
Nov 02 01:29:05.866923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:05.866887    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:06.015998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:06.015965    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:07.060316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:07.059367    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:07.232432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:07.232386    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:07.428370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:07.358227    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:07.436345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:07.404078    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:29:07.532419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:07.532371    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:08.811427 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:08.811396    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:09.615299 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.615264    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"]
Nov 02 01:29:09.615853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.615837    4851 cpu_manager.go:414] "containerMap: removing container" podUID="3119f15b-88ea-41ab-a4fa-c4a128051041" containerName="revoke-keys"
Nov 02 01:29:09.615908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.615856    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3119f15b-88ea-41ab-a4fa-c4a128051041" containerName="revoke-keys"
Nov 02 01:29:09.615908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.615889    4851 cpu_manager.go:414] "containerMap: removing container" podUID="f6c35bed-145e-4b2c-88b2-0c5963d2df88" containerName="revoke-keys"
Nov 02 01:29:09.615908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.615895    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c35bed-145e-4b2c-88b2-0c5963d2df88" containerName="revoke-keys"
Nov 02 01:29:09.616026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.616015    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c35bed-145e-4b2c-88b2-0c5963d2df88" containerName="revoke-keys"
Nov 02 01:29:09.616067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.616027    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="3119f15b-88ea-41ab-a4fa-c4a128051041" containerName="revoke-keys"
Nov 02 01:29:09.618328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.618282    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.623606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.623583    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:29:09.623885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.623856    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:29:09.623995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.623889    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:29:09.623995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.623886    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:29:09.629829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.629806    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"]
Nov 02 01:29:09.715192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.715162    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-930f803f-w5x55\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.715408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.715207    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-930f803f-w5x55\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.715408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.715357    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cncgb\" (UniqueName: \"kubernetes.io/projected/b80f66a8-8962-497c-ad8e-c6e8b4660713-kube-api-access-cncgb\") pod \"maas-api-revoke-sub-930f803f-w5x55\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.816645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.816615    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cncgb\" (UniqueName: \"kubernetes.io/projected/b80f66a8-8962-497c-ad8e-c6e8b4660713-kube-api-access-cncgb\") pod \"maas-api-revoke-sub-930f803f-w5x55\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.817100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.816883    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-930f803f-w5x55\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.817100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.816926    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-930f803f-w5x55\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.817654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.817630    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-930f803f-w5x55\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.819388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.819369    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-930f803f-w5x55\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.830875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.830854    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cncgb\" (UniqueName: \"kubernetes.io/projected/b80f66a8-8962-497c-ad8e-c6e8b4660713-kube-api-access-cncgb\") pod \"maas-api-revoke-sub-930f803f-w5x55\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.932795 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.932769    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:09.939541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:09.939519    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:10.315715 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:10.315680    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"]
Nov 02 01:29:10.315870 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:10.315808    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80f66a8_8962_497c_ad8e_c6e8b4660713.slice/crio-93ce97e4bee60f851bf2ed1831d1f2eee3e890f6c6f8c9878d0b56dc3f9be106 WatchSource:0}: Error finding container 93ce97e4bee60f851bf2ed1831d1f2eee3e890f6c6f8c9878d0b56dc3f9be106: Status 404 returned error can't find the container with id 93ce97e4bee60f851bf2ed1831d1f2eee3e890f6c6f8c9878d0b56dc3f9be106
Nov 02 01:29:10.531695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:10.531660    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55" event={"ID":"b80f66a8-8962-497c-ad8e-c6e8b4660713","Type":"ContainerStarted","Data":"93ce97e4bee60f851bf2ed1831d1f2eee3e890f6c6f8c9878d0b56dc3f9be106"}
Nov 02 01:29:10.984554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:10.984523    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:11.094730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.094699    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"]
Nov 02 01:29:11.097815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.097799    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.108244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.107951    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"]
Nov 02 01:29:11.166527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.166497    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e82dc1-pq9zb\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.166951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.166922    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6z4\" (UniqueName: \"kubernetes.io/projected/52f8a717-4e43-4d09-99a2-97d07bdad40d-kube-api-access-6d6z4\") pod \"maas-api-revoke-sub-13e82dc1-pq9zb\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.167133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.167114    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e82dc1-pq9zb\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.268751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.268666    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e82dc1-pq9zb\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.268751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.268721    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e82dc1-pq9zb\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.268957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.268854    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6z4\" (UniqueName: \"kubernetes.io/projected/52f8a717-4e43-4d09-99a2-97d07bdad40d-kube-api-access-6d6z4\") pod \"maas-api-revoke-sub-13e82dc1-pq9zb\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.269403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.269381    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e82dc1-pq9zb\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.271083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.271060    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e82dc1-pq9zb\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.281139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.281117    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6z4\" (UniqueName: \"kubernetes.io/projected/52f8a717-4e43-4d09-99a2-97d07bdad40d-kube-api-access-6d6z4\") pod \"maas-api-revoke-sub-13e82dc1-pq9zb\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.414702 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.414658    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:11.552266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.552174    4851 generic.go:356] "Generic (PLEG): container finished" podID="b80f66a8-8962-497c-ad8e-c6e8b4660713" containerID="f529cc7c5c46efb25fa8b794ace2dded2b7b3b14a75043030d7974619f42f04d" exitCode=0
Nov 02 01:29:11.552467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.552257    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55" event={"ID":"b80f66a8-8962-497c-ad8e-c6e8b4660713","Type":"ContainerDied","Data":"f529cc7c5c46efb25fa8b794ace2dded2b7b3b14a75043030d7974619f42f04d"}
Nov 02 01:29:11.652080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.652040    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"]
Nov 02 01:29:11.656586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.656565    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:11.673009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.669041    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"]
Nov 02 01:29:11.706996 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:11.706971    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f8a717_4e43_4d09_99a2_97d07bdad40d.slice/crio-69b63b5c6a335fa49de9e8a56e91a2362ae9252144bfcf49edfa13c571f86f6b WatchSource:0}: Error finding container 69b63b5c6a335fa49de9e8a56e91a2362ae9252144bfcf49edfa13c571f86f6b: Status 404 returned error can't find the container with id 69b63b5c6a335fa49de9e8a56e91a2362ae9252144bfcf49edfa13c571f86f6b
Nov 02 01:29:11.707771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.707747    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"]
Nov 02 01:29:11.787180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.787150    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcrs\" (UniqueName: \"kubernetes.io/projected/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-kube-api-access-gmcrs\") pod \"maas-api-revoke-sub-2721d457-vgtss\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:11.787392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.787191    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2721d457-vgtss\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:11.787392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.787380    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2721d457-vgtss\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:11.887658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.887572    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:11.888231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.888210    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcrs\" (UniqueName: \"kubernetes.io/projected/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-kube-api-access-gmcrs\") pod \"maas-api-revoke-sub-2721d457-vgtss\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:11.888324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.888241    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2721d457-vgtss\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:11.888395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.888333    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2721d457-vgtss\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:11.888959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.888941    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2721d457-vgtss\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:11.890784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.890760    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2721d457-vgtss\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:11.900445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.900421    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcrs\" (UniqueName: \"kubernetes.io/projected/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-kube-api-access-gmcrs\") pod \"maas-api-revoke-sub-2721d457-vgtss\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:11.999315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:11.996603    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:12.361768 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:12.361727    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028dec5f_8122_4d7e_8b8f_6cf9d04d6f4c.slice/crio-a6a905cacb0318282cdf5b4bb4b3c26b3c0017df9f2805026851d3e3584d7d55 WatchSource:0}: Error finding container a6a905cacb0318282cdf5b4bb4b3c26b3c0017df9f2805026851d3e3584d7d55: Status 404 returned error can't find the container with id a6a905cacb0318282cdf5b4bb4b3c26b3c0017df9f2805026851d3e3584d7d55
Nov 02 01:29:12.363140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.363108    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"]
Nov 02 01:29:12.563497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.563457    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss" event={"ID":"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c","Type":"ContainerStarted","Data":"a6a905cacb0318282cdf5b4bb4b3c26b3c0017df9f2805026851d3e3584d7d55"}
Nov 02 01:29:12.565187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.565156    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb" event={"ID":"52f8a717-4e43-4d09-99a2-97d07bdad40d","Type":"ContainerStarted","Data":"701a651c2ff0db01aa7e49d5e4e9ea83c27330a5086f204a883c1f4fdfab43b7"}
Nov 02 01:29:12.565375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.565212    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb" event={"ID":"52f8a717-4e43-4d09-99a2-97d07bdad40d","Type":"ContainerStarted","Data":"69b63b5c6a335fa49de9e8a56e91a2362ae9252144bfcf49edfa13c571f86f6b"}
Nov 02 01:29:12.586424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.586372    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb" podStartSLOduration=1.5863551089999999 podStartE2EDuration="1.586355109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:12.583181334 +0000 UTC m=+857.888114159" watchObservedRunningTime="2026-11-02 01:29:12.586355109 +0000 UTC m=+857.891287935"
Nov 02 01:29:12.706652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.706612    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:12.819838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.819805    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:12.909544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.909510    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-xks-ca\") pod \"b80f66a8-8962-497c-ad8e-c6e8b4660713\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") "
Nov 02 01:29:12.909716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.909596    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80f66a8-8962-497c-ad8e-c6e8b4660713-kube-api-access-cncgb\" (UniqueName: \"kubernetes.io/projected/b80f66a8-8962-497c-ad8e-c6e8b4660713-kube-api-access-cncgb\") pod \"b80f66a8-8962-497c-ad8e-c6e8b4660713\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") "
Nov 02 01:29:12.909716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.909672    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-openshift-ca\") pod \"b80f66a8-8962-497c-ad8e-c6e8b4660713\" (UID: \"b80f66a8-8962-497c-ad8e-c6e8b4660713\") "
Nov 02 01:29:12.910099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.910070    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-openshift-ca" pod "b80f66a8-8962-497c-ad8e-c6e8b4660713" (UID: "b80f66a8-8962-497c-ad8e-c6e8b4660713"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:12.910276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.910257    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:12.911987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.911963    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-xks-ca" pod "b80f66a8-8962-497c-ad8e-c6e8b4660713" (UID: "b80f66a8-8962-497c-ad8e-c6e8b4660713"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:12.912064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:12.912040    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80f66a8-8962-497c-ad8e-c6e8b4660713-kube-api-access-cncgb" pod "b80f66a8-8962-497c-ad8e-c6e8b4660713" (UID: "b80f66a8-8962-497c-ad8e-c6e8b4660713"). InnerVolumeSpecName "kube-api-access-cncgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:13.011963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.011869    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b80f66a8-8962-497c-ad8e-c6e8b4660713-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:13.011963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.011899    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cncgb\" (UniqueName: \"kubernetes.io/projected/b80f66a8-8962-497c-ad8e-c6e8b4660713-kube-api-access-cncgb\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:13.359222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.359194    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"]
Nov 02 01:29:13.359834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.359821    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b80f66a8-8962-497c-ad8e-c6e8b4660713" containerName="revoke-keys"
Nov 02 01:29:13.359898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.359837    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80f66a8-8962-497c-ad8e-c6e8b4660713" containerName="revoke-keys"
Nov 02 01:29:13.359969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.359960    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80f66a8-8962-497c-ad8e-c6e8b4660713" containerName="revoke-keys"
Nov 02 01:29:13.362377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.362363    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:13.383434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.378277    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"]
Nov 02 01:29:13.520693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.520659    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353d7753-8slxj\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:13.520849 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.520774    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vps2l\" (UniqueName: \"kubernetes.io/projected/75d6724d-9aea-4f0e-96c8-24b977eff876-kube-api-access-vps2l\") pod \"maas-api-revoke-sub-353d7753-8slxj\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:13.520849 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.520835    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353d7753-8slxj\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:13.573994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.573962    4851 generic.go:356] "Generic (PLEG): container finished" podID="028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c" containerID="6e1869f8ac8193dab9a2beb9230d9bcfddb08401e2cb93ed53f974f348baa064" exitCode=0
Nov 02 01:29:13.574210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.574050    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss" event={"ID":"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c","Type":"ContainerDied","Data":"6e1869f8ac8193dab9a2beb9230d9bcfddb08401e2cb93ed53f974f348baa064"}
Nov 02 01:29:13.575663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.575642    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55" event={"ID":"b80f66a8-8962-497c-ad8e-c6e8b4660713","Type":"ContainerDied","Data":"93ce97e4bee60f851bf2ed1831d1f2eee3e890f6c6f8c9878d0b56dc3f9be106"}
Nov 02 01:29:13.575663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.575655    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"
Nov 02 01:29:13.575825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.575684    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93ce97e4bee60f851bf2ed1831d1f2eee3e890f6c6f8c9878d0b56dc3f9be106"
Nov 02 01:29:13.577430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.577409    4851 generic.go:356] "Generic (PLEG): container finished" podID="52f8a717-4e43-4d09-99a2-97d07bdad40d" containerID="701a651c2ff0db01aa7e49d5e4e9ea83c27330a5086f204a883c1f4fdfab43b7" exitCode=0
Nov 02 01:29:13.577527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.577496    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb" event={"ID":"52f8a717-4e43-4d09-99a2-97d07bdad40d","Type":"ContainerDied","Data":"701a651c2ff0db01aa7e49d5e4e9ea83c27330a5086f204a883c1f4fdfab43b7"}
Nov 02 01:29:13.606758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.606730    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:13.653159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.653124    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353d7753-8slxj\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:13.683319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.679511    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vps2l\" (UniqueName: \"kubernetes.io/projected/75d6724d-9aea-4f0e-96c8-24b977eff876-kube-api-access-vps2l\") pod \"maas-api-revoke-sub-353d7753-8slxj\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:13.683319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.679633    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353d7753-8slxj\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:13.683319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.654624    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353d7753-8slxj\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:13.693190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.693157    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353d7753-8slxj\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:13.720778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:13.720750    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vps2l\" (UniqueName: \"kubernetes.io/projected/75d6724d-9aea-4f0e-96c8-24b977eff876-kube-api-access-vps2l\") pod \"maas-api-revoke-sub-353d7753-8slxj\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:14.018468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.018435    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:14.299017 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:14.298983    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d6724d_9aea_4f0e_96c8_24b977eff876.slice/crio-5b785cb6cbb806deb9e90e0b42b316af037f621aff9db6654e4777d82765b6d9 WatchSource:0}: Error finding container 5b785cb6cbb806deb9e90e0b42b316af037f621aff9db6654e4777d82765b6d9: Status 404 returned error can't find the container with id 5b785cb6cbb806deb9e90e0b42b316af037f621aff9db6654e4777d82765b6d9
Nov 02 01:29:14.300141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.300115    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"]
Nov 02 01:29:14.394579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.393811    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:14.586753 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.586677    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj" event={"ID":"75d6724d-9aea-4f0e-96c8-24b977eff876","Type":"ContainerStarted","Data":"5b785cb6cbb806deb9e90e0b42b316af037f621aff9db6654e4777d82765b6d9"}
Nov 02 01:29:14.756078 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.756052    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:14.787670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.787646    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:14.820076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.820045    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-xks-ca\") pod \"52f8a717-4e43-4d09-99a2-97d07bdad40d\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") "
Nov 02 01:29:14.820234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.820108    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f8a717-4e43-4d09-99a2-97d07bdad40d-kube-api-access-6d6z4\" (UniqueName: \"kubernetes.io/projected/52f8a717-4e43-4d09-99a2-97d07bdad40d-kube-api-access-6d6z4\") pod \"52f8a717-4e43-4d09-99a2-97d07bdad40d\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") "
Nov 02 01:29:14.820476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.820456    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-openshift-ca\") pod \"52f8a717-4e43-4d09-99a2-97d07bdad40d\" (UID: \"52f8a717-4e43-4d09-99a2-97d07bdad40d\") "
Nov 02 01:29:14.821219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.821196    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-openshift-ca" pod "52f8a717-4e43-4d09-99a2-97d07bdad40d" (UID: "52f8a717-4e43-4d09-99a2-97d07bdad40d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:14.823272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.823246    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f8a717-4e43-4d09-99a2-97d07bdad40d-kube-api-access-6d6z4" pod "52f8a717-4e43-4d09-99a2-97d07bdad40d" (UID: "52f8a717-4e43-4d09-99a2-97d07bdad40d"). InnerVolumeSpecName "kube-api-access-6d6z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:14.824358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.824339    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-xks-ca" pod "52f8a717-4e43-4d09-99a2-97d07bdad40d" (UID: "52f8a717-4e43-4d09-99a2-97d07bdad40d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:14.921953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.921926    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-openshift-ca\") pod \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") "
Nov 02 01:29:14.922073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.921981    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-kube-api-access-gmcrs\" (UniqueName: \"kubernetes.io/projected/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-kube-api-access-gmcrs\") pod \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") "
Nov 02 01:29:14.922124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.922071    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-xks-ca\") pod \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\" (UID: \"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c\") "
Nov 02 01:29:14.922384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.922357    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-openshift-ca" pod "028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c" (UID: "028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:14.922951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.922891    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:14.922951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.922919    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f8a717-4e43-4d09-99a2-97d07bdad40d-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:14.922951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.922934    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d6z4\" (UniqueName: \"kubernetes.io/projected/52f8a717-4e43-4d09-99a2-97d07bdad40d-kube-api-access-6d6z4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:14.922951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.922949    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:14.924367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.924263    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-xks-ca" pod "028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c" (UID: "028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:14.924456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:14.924393    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-kube-api-access-gmcrs" pod "028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c" (UID: "028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c"). InnerVolumeSpecName "kube-api-access-gmcrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:15.024466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.024429    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:15.024466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.024464    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmcrs\" (UniqueName: \"kubernetes.io/projected/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c-kube-api-access-gmcrs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:15.215275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.215237    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:15.594905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.594837    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"
Nov 02 01:29:15.595068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.594835    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb" event={"ID":"52f8a717-4e43-4d09-99a2-97d07bdad40d","Type":"ContainerDied","Data":"69b63b5c6a335fa49de9e8a56e91a2362ae9252144bfcf49edfa13c571f86f6b"}
Nov 02 01:29:15.595068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.594967    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69b63b5c6a335fa49de9e8a56e91a2362ae9252144bfcf49edfa13c571f86f6b"
Nov 02 01:29:15.596645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.596620    4851 generic.go:356] "Generic (PLEG): container finished" podID="75d6724d-9aea-4f0e-96c8-24b977eff876" containerID="f312c298ac3a54e7c1006b2ba8732b7b24ea76285681f620a84f6aac29ad4999" exitCode=0
Nov 02 01:29:15.596779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.596705    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj" event={"ID":"75d6724d-9aea-4f0e-96c8-24b977eff876","Type":"ContainerDied","Data":"f312c298ac3a54e7c1006b2ba8732b7b24ea76285681f620a84f6aac29ad4999"}
Nov 02 01:29:15.598358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.598337    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss" event={"ID":"028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c","Type":"ContainerDied","Data":"a6a905cacb0318282cdf5b4bb4b3c26b3c0017df9f2805026851d3e3584d7d55"}
Nov 02 01:29:15.598502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.598361    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"
Nov 02 01:29:15.598502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.598377    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6a905cacb0318282cdf5b4bb4b3c26b3c0017df9f2805026851d3e3584d7d55"
Nov 02 01:29:15.934462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:15.934431    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:16.716888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:16.716848    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:16.841699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:16.841667    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:16.987015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:16.986985    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-xks-ca\") pod \"75d6724d-9aea-4f0e-96c8-24b977eff876\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") "
Nov 02 01:29:16.987188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:16.987114    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d6724d-9aea-4f0e-96c8-24b977eff876-kube-api-access-vps2l\" (UniqueName: \"kubernetes.io/projected/75d6724d-9aea-4f0e-96c8-24b977eff876-kube-api-access-vps2l\") pod \"75d6724d-9aea-4f0e-96c8-24b977eff876\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") "
Nov 02 01:29:16.987239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:16.987220    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-openshift-ca\") pod \"75d6724d-9aea-4f0e-96c8-24b977eff876\" (UID: \"75d6724d-9aea-4f0e-96c8-24b977eff876\") "
Nov 02 01:29:16.987646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:16.987617    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-openshift-ca" pod "75d6724d-9aea-4f0e-96c8-24b977eff876" (UID: "75d6724d-9aea-4f0e-96c8-24b977eff876"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:16.989247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:16.989227    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-xks-ca" pod "75d6724d-9aea-4f0e-96c8-24b977eff876" (UID: "75d6724d-9aea-4f0e-96c8-24b977eff876"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:16.989411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:16.989394    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d6724d-9aea-4f0e-96c8-24b977eff876-kube-api-access-vps2l" pod "75d6724d-9aea-4f0e-96c8-24b977eff876" (UID: "75d6724d-9aea-4f0e-96c8-24b977eff876"). InnerVolumeSpecName "kube-api-access-vps2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:17.089091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:17.089058    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vps2l\" (UniqueName: \"kubernetes.io/projected/75d6724d-9aea-4f0e-96c8-24b977eff876-kube-api-access-vps2l\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:17.089091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:17.089088    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:17.089091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:17.089098    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d6724d-9aea-4f0e-96c8-24b977eff876-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:17.422250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:17.422212    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:17.615933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:17.615904    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"
Nov 02 01:29:17.615933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:17.615934    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj" event={"ID":"75d6724d-9aea-4f0e-96c8-24b977eff876","Type":"ContainerDied","Data":"5b785cb6cbb806deb9e90e0b42b316af037f621aff9db6654e4777d82765b6d9"}
Nov 02 01:29:17.616133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:17.615986    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b785cb6cbb806deb9e90e0b42b316af037f621aff9db6654e4777d82765b6d9"
Nov 02 01:29:17.656390 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:29:17.656347    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:29:18.155764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:18.155729    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:18.812904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:18.812874    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:19.619179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:19.619146    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:20.355732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:20.355691    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:21.127741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:21.127705    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:21.865210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:21.865178    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:22.689079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:22.689048    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:23.040769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.040692    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"]
Nov 02 01:29:23.041319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.041282    4851 cpu_manager.go:414] "containerMap: removing container" podUID="75d6724d-9aea-4f0e-96c8-24b977eff876" containerName="revoke-keys"
Nov 02 01:29:23.041319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.041312    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d6724d-9aea-4f0e-96c8-24b977eff876" containerName="revoke-keys"
Nov 02 01:29:23.041389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.041337    4851 cpu_manager.go:414] "containerMap: removing container" podUID="028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c" containerName="revoke-keys"
Nov 02 01:29:23.041389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.041342    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c" containerName="revoke-keys"
Nov 02 01:29:23.041389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.041352    4851 cpu_manager.go:414] "containerMap: removing container" podUID="52f8a717-4e43-4d09-99a2-97d07bdad40d" containerName="revoke-keys"
Nov 02 01:29:23.041389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.041358    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f8a717-4e43-4d09-99a2-97d07bdad40d" containerName="revoke-keys"
Nov 02 01:29:23.041520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.041445    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d6724d-9aea-4f0e-96c8-24b977eff876" containerName="revoke-keys"
Nov 02 01:29:23.041520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.041454    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f8a717-4e43-4d09-99a2-97d07bdad40d" containerName="revoke-keys"
Nov 02 01:29:23.041520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.041465    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c" containerName="revoke-keys"
Nov 02 01:29:23.043894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.043877    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.047878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.047852    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:29:23.048732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.048715    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:29:23.048881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.048743    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:29:23.048881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.048741    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:29:23.053330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.053271    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"]
Nov 02 01:29:23.077046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.077013    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvlnf\" (UniqueName: \"kubernetes.io/projected/b0ae183c-f86b-4560-adca-9430c3f39eeb-kube-api-access-nvlnf\") pod \"maas-api-revoke-sub-2e13dc1b-zxzkz\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.077258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.077239    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e13dc1b-zxzkz\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.077358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.077281    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e13dc1b-zxzkz\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.190315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.186566    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvlnf\" (UniqueName: \"kubernetes.io/projected/b0ae183c-f86b-4560-adca-9430c3f39eeb-kube-api-access-nvlnf\") pod \"maas-api-revoke-sub-2e13dc1b-zxzkz\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.190315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.186780    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e13dc1b-zxzkz\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.190315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.186819    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e13dc1b-zxzkz\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.223316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.199658    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e13dc1b-zxzkz\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.223570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.213751    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e13dc1b-zxzkz\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.228965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.228935    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvlnf\" (UniqueName: \"kubernetes.io/projected/b0ae183c-f86b-4560-adca-9430c3f39eeb-kube-api-access-nvlnf\") pod \"maas-api-revoke-sub-2e13dc1b-zxzkz\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.374372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.368988    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:23.856732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:23.856680    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:24.185252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:24.185212    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"]
Nov 02 01:29:24.577317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:24.549680    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.031278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.031223    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.033954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.033744    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.034242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.034222    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375a3837-853f-4bb4-b078-b2116e0d80f0-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.034473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.034433    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.034715 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.034694    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrlt5\" (UniqueName: \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-kube-api-access-jrlt5\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.035014 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.034996    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.035254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.035235    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.035435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.035417    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.039332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.035577    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.113425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.113367    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:25.229282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.229243    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.230422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.230398    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.230606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.230588    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.230787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.230769    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375a3837-853f-4bb4-b078-b2116e0d80f0-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.230931 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.230914    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.231063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.231045    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrlt5\" (UniqueName: \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-kube-api-access-jrlt5\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.231384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.231361    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.231773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.231752    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.235533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.235490    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.261343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.237979    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.261343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.238251    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.261343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.238771    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.261343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.239994    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375a3837-853f-4bb4-b078-b2116e0d80f0-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.261343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.240249    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.261343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.243665    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:25.261343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.233634    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:25.294421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.289997    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:25.317324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.302480    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"]
Nov 02 01:29:25.317324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.302594    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"]
Nov 02 01:29:25.317324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.306552    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.324372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.320369    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.391319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.348402    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.391319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.375483    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz" event={"ID":"b0ae183c-f86b-4560-adca-9430c3f39eeb","Type":"ContainerStarted","Data":"ed0ed057ed37fd518a2dbb46e621d2817ea812fdbf61cc2b269e480cdf6887c3"}
Nov 02 01:29:25.391319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.375614    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.415317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.401788    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrlt5\" (UniqueName: \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-kube-api-access-jrlt5\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.415317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.405849    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"]
Nov 02 01:29:25.447674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.447641    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490440    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490486    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6p6\" (UniqueName: \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-kube-api-access-nb6p6\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490518    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adc04e2-bg9ww\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490563    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s95kh\" (UniqueName: \"kubernetes.io/projected/c5a39c85-e039-4c53-896d-f81e4754917c-kube-api-access-s95kh\") pod \"maas-api-revoke-sub-0adc04e2-bg9ww\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490586    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490618    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504edd1e-79af-4ceb-85a7-507a8f6de614-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490716    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adc04e2-bg9ww\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490781    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490920    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490968    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.490997    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.501693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.491031    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.510105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.510058    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"]
Nov 02 01:29:25.513645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.513609    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:25.514161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.514135    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:29:25.522699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.518094    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:29:25.597510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.597473    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.597814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.597791    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.598773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.598742    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.609830    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.609902    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.609941    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6p6\" (UniqueName: \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-kube-api-access-nb6p6\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.609976    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adc04e2-bg9ww\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.610038    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s95kh\" (UniqueName: \"kubernetes.io/projected/c5a39c85-e039-4c53-896d-f81e4754917c-kube-api-access-s95kh\") pod \"maas-api-revoke-sub-0adc04e2-bg9ww\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.610065    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.610098    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504edd1e-79af-4ceb-85a7-507a8f6de614-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.610255    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adc04e2-bg9ww\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.610334    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.610836    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.598383    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.611105    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.611353    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.612999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.612378    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adc04e2-bg9ww\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:25.617250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.614463    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504edd1e-79af-4ceb-85a7-507a8f6de614-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.625673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.624996    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"]
Nov 02 01:29:25.645684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.644554    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:25.646590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.646363    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:25.650324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.646985    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.681317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.680557    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:25.683322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.682017    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6p6\" (UniqueName: \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-kube-api-access-nb6p6\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.711980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.709735    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.719951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.719925    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.725707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.722698    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adc04e2-bg9ww\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:25.742711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.741447    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:25.750957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.750925    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s95kh\" (UniqueName: \"kubernetes.io/projected/c5a39c85-e039-4c53-896d-f81e4754917c-kube-api-access-s95kh\") pod \"maas-api-revoke-sub-0adc04e2-bg9ww\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:25.791375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.784436    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:25.792189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.792087    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"]
Nov 02 01:29:25.815329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.814761    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:25.905363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:25.905321    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:26.047626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.047537    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:26.323143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.293217    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:29:26.323143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.294104    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:29:26.339513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.323518    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:26.340443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.340407    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:26.341782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.341753    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:29:26.341896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.341823    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:29:26.365348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.343011    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:29:26.530528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.528437    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:26.530528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.529100    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:29:26.664335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.661422    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:26.679486 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.665735    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:29:26.758575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.746077    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:26.832939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.799585    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:26.832939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.811027    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:29:26.864396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.862867    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:26.949899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.948413    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Nov 02 01:29:26.949899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:26.948730    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:29:27.411865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.383647    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:27.588318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.585140    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:29:27.614319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.608219    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:29:27.652626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.652557    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:27.682382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.681969    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:27.715585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.715527    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:27.796822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.716897    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:29:27.798793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.729335    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:27.799144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.799110    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:27.826638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.813182    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:27.826638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.819764    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:27.835318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.833711    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:29:27.835617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.835587    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:27.874424 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:29:27.868137    4851 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.029s"
Nov 02 01:29:27.874424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.868185    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz" event={"ID":"b0ae183c-f86b-4560-adca-9430c3f39eeb","Type":"ContainerStarted","Data":"9da58647699082fd7610ac4abe6b56284479ffae3551ec96f224ca9b6abfd8b2"}
Nov 02 01:29:27.874424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.868484    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:29:27.874424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.868552    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:29:27.874424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.868619    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:29:27.874424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.868688    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:29:27.874424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.868738    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:29:27.874424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.868783    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:29:27.874424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.868802    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"]
Nov 02 01:29:27.874424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.868877    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:29:27.899721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.898742    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:29:27.904756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.903539    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:29:27.905875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.905444    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:29:27.905875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.905809    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:29:27.922552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.922526    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:29:27.942462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.923209    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:27.942462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.923345    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:27.942462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.923445    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:27.947918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.947069    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"]
Nov 02 01:29:27.961357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.961260    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz" podStartSLOduration=4.961239056 podStartE2EDuration="4.961239056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:27.887424645 +0000 UTC m=+873.192357472" watchObservedRunningTime="2026-11-02 01:29:27.961239056 +0000 UTC m=+873.266171884"
Nov 02 01:29:27.964203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.964172    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" event={"ID":"504edd1e-79af-4ceb-85a7-507a8f6de614","Type":"ContainerStarted","Data":"c41d8fa415f42ddcffc54ada60473b2347db8c2dac96c2fa137f6aab319c02e8"}
Nov 02 01:29:27.986814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.986781    4851 generic.go:356] "Generic (PLEG): container finished" podID="b0ae183c-f86b-4560-adca-9430c3f39eeb" containerID="9da58647699082fd7610ac4abe6b56284479ffae3551ec96f224ca9b6abfd8b2" exitCode=0
Nov 02 01:29:27.986987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.986936    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz" event={"ID":"b0ae183c-f86b-4560-adca-9430c3f39eeb","Type":"ContainerDied","Data":"9da58647699082fd7610ac4abe6b56284479ffae3551ec96f224ca9b6abfd8b2"}
Nov 02 01:29:27.989483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:27.989461    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" event={"ID":"375a3837-853f-4bb4-b078-b2116e0d80f0","Type":"ContainerStarted","Data":"0796d1a3342b6433d7b330beb7980336699114634c7b247d2485cd7c79895c63"}
Nov 02 01:29:28.171597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.166730    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"]
Nov 02 01:29:28.205875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.205792    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"]
Nov 02 01:29:28.214246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.214224    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.215394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.215370    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"]
Nov 02 01:29:28.227116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.227088    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6407b46-cmsk7\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.227281 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.227130    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqxww\" (UniqueName: \"kubernetes.io/projected/362c6a8e-907e-4aae-b272-412bbe0c848b-kube-api-access-pqxww\") pod \"maas-api-revoke-sub-f6407b46-cmsk7\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.227394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.227370    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6407b46-cmsk7\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.243897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.243770    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:28.390799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.390681    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6407b46-cmsk7\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.390985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.390962    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6407b46-cmsk7\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.391170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.391151    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxww\" (UniqueName: \"kubernetes.io/projected/362c6a8e-907e-4aae-b272-412bbe0c848b-kube-api-access-pqxww\") pod \"maas-api-revoke-sub-f6407b46-cmsk7\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.394328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.392591    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6407b46-cmsk7\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.401894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.401866    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6407b46-cmsk7\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.405245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.405189    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxww\" (UniqueName: \"kubernetes.io/projected/362c6a8e-907e-4aae-b272-412bbe0c848b-kube-api-access-pqxww\") pod \"maas-api-revoke-sub-f6407b46-cmsk7\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.553313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.553269    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:28.744409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.744374    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"]
Nov 02 01:29:28.760963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.760937    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:28.761966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.761918    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"]
Nov 02 01:29:28.846306 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:28.846255    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362c6a8e_907e_4aae_b272_412bbe0c848b.slice/crio-bb037ffca2857739874c4342dfcc88f70beea7e9f1ac7c6e563dd07e175f32b4 WatchSource:0}: Error finding container bb037ffca2857739874c4342dfcc88f70beea7e9f1ac7c6e563dd07e175f32b4: Status 404 returned error can't find the container with id bb037ffca2857739874c4342dfcc88f70beea7e9f1ac7c6e563dd07e175f32b4
Nov 02 01:29:28.851116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.851042    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"]
Nov 02 01:29:28.903411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.903373    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6810f982-sx8vj\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:28.903624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.903606    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4bm\" (UniqueName: \"kubernetes.io/projected/c7a2d739-07d2-437e-bd57-806e0525ad24-kube-api-access-kk4bm\") pod \"maas-api-revoke-sub-6810f982-sx8vj\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:28.903756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:28.903743    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6810f982-sx8vj\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:29.005382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.005349    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6810f982-sx8vj\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:29.005735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.005711    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4bm\" (UniqueName: \"kubernetes.io/projected/c7a2d739-07d2-437e-bd57-806e0525ad24-kube-api-access-kk4bm\") pod \"maas-api-revoke-sub-6810f982-sx8vj\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:29.005827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.005807    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6810f982-sx8vj\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:29.007205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.007178    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6810f982-sx8vj\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:29.009961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.009940    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6810f982-sx8vj\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:29.012995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.012968    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" event={"ID":"504edd1e-79af-4ceb-85a7-507a8f6de614","Type":"ContainerStarted","Data":"c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24"}
Nov 02 01:29:29.015461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.015438    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" event={"ID":"375a3837-853f-4bb4-b078-b2116e0d80f0","Type":"ContainerStarted","Data":"e285dfc421df0740b0f75a7cf666e3e6083642ba163514178fe873ead9903dee"}
Nov 02 01:29:29.017011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.016987    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7" event={"ID":"362c6a8e-907e-4aae-b272-412bbe0c848b","Type":"ContainerStarted","Data":"bb037ffca2857739874c4342dfcc88f70beea7e9f1ac7c6e563dd07e175f32b4"}
Nov 02 01:29:29.018982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.018909    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww" event={"ID":"c5a39c85-e039-4c53-896d-f81e4754917c","Type":"ContainerStarted","Data":"9b47e6449d6a58080d921547dcc5fc4de1bceeb5c8c44cd7e39e24e7e45e87e0"}
Nov 02 01:29:29.018982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.018937    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww" event={"ID":"c5a39c85-e039-4c53-896d-f81e4754917c","Type":"ContainerStarted","Data":"fcbc50d0f56ef538a6b13b9cf9187d0e2bc5df9dd5aea665a8e5586a3e9f4f45"}
Nov 02 01:29:29.020214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.020192    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4bm\" (UniqueName: \"kubernetes.io/projected/c7a2d739-07d2-437e-bd57-806e0525ad24-kube-api-access-kk4bm\") pod \"maas-api-revoke-sub-6810f982-sx8vj\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:29.085337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.083533    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:29.110586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.110538    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" podStartSLOduration=5.1105237 podStartE2EDuration="5.1105237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:29.07721651 +0000 UTC m=+874.382149337" watchObservedRunningTime="2026-11-02 01:29:29.1105237 +0000 UTC m=+874.415456526"
Nov 02 01:29:29.129450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.129386    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" podStartSLOduration=5.129372556 podStartE2EDuration="5.129372556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:29.109769883 +0000 UTC m=+874.414702709" watchObservedRunningTime="2026-11-02 01:29:29.129372556 +0000 UTC m=+874.434305382"
Nov 02 01:29:29.196698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.196613    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww" podStartSLOduration=5.19659406 podStartE2EDuration="5.19659406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:29.128089124 +0000 UTC m=+874.433021951" watchObservedRunningTime="2026-11-02 01:29:29.19659406 +0000 UTC m=+874.501526888"
Nov 02 01:29:29.312805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.312720    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:29.518696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.518666    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:29.591913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.591863    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-xks-ca\") pod \"b0ae183c-f86b-4560-adca-9430c3f39eeb\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") "
Nov 02 01:29:29.592549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.592523    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ae183c-f86b-4560-adca-9430c3f39eeb-kube-api-access-nvlnf\" (UniqueName: \"kubernetes.io/projected/b0ae183c-f86b-4560-adca-9430c3f39eeb-kube-api-access-nvlnf\") pod \"b0ae183c-f86b-4560-adca-9430c3f39eeb\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") "
Nov 02 01:29:29.592894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.592867    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-openshift-ca\") pod \"b0ae183c-f86b-4560-adca-9430c3f39eeb\" (UID: \"b0ae183c-f86b-4560-adca-9430c3f39eeb\") "
Nov 02 01:29:29.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.594173    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-openshift-ca" pod "b0ae183c-f86b-4560-adca-9430c3f39eeb" (UID: "b0ae183c-f86b-4560-adca-9430c3f39eeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:29.596911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.596633    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ae183c-f86b-4560-adca-9430c3f39eeb-kube-api-access-nvlnf" pod "b0ae183c-f86b-4560-adca-9430c3f39eeb" (UID: "b0ae183c-f86b-4560-adca-9430c3f39eeb"). InnerVolumeSpecName "kube-api-access-nvlnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:29.596911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.596758    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-xks-ca" pod "b0ae183c-f86b-4560-adca-9430c3f39eeb" (UID: "b0ae183c-f86b-4560-adca-9430c3f39eeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:29.681565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.679879    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:29.681565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.680327    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:29:29.681565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.680379    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 01:29:29.691549 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:29.691510    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a2d739_07d2_437e_bd57_806e0525ad24.slice/crio-6e384bd7606ab7ec4927d6ea0b4431595496c62238180bff83f0dec10a2490b0 WatchSource:0}: Error finding container 6e384bd7606ab7ec4927d6ea0b4431595496c62238180bff83f0dec10a2490b0: Status 404 returned error can't find the container with id 6e384bd7606ab7ec4927d6ea0b4431595496c62238180bff83f0dec10a2490b0
Nov 02 01:29:29.692001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.691970    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"]
Nov 02 01:29:29.694613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.694411    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvlnf\" (UniqueName: \"kubernetes.io/projected/b0ae183c-f86b-4560-adca-9430c3f39eeb-kube-api-access-nvlnf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:29.694613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.694429    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:29.694613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.694440    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ae183c-f86b-4560-adca-9430c3f39eeb-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:29.754317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.743338    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:29.764326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.758414    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:29:29.764326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:29.758489    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 01:29:30.003947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.003858    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"]
Nov 02 01:29:30.004905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.004883    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b0ae183c-f86b-4560-adca-9430c3f39eeb" containerName="revoke-keys"
Nov 02 01:29:30.004968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.004910    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ae183c-f86b-4560-adca-9430c3f39eeb" containerName="revoke-keys"
Nov 02 01:29:30.005100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.005089    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ae183c-f86b-4560-adca-9430c3f39eeb" containerName="revoke-keys"
Nov 02 01:29:30.009605 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.009583    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.010261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.010238    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"]
Nov 02 01:29:30.100458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.100425    4851 generic.go:356] "Generic (PLEG): container finished" podID="c5a39c85-e039-4c53-896d-f81e4754917c" containerID="9b47e6449d6a58080d921547dcc5fc4de1bceeb5c8c44cd7e39e24e7e45e87e0" exitCode=0
Nov 02 01:29:30.100685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.100660    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv762\" (UniqueName: \"kubernetes.io/projected/29356a60-2eeb-438b-a57b-a589b3faf0f8-kube-api-access-pv762\") pod \"maas-api-revoke-sub-b4e128ad-twpqk\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.100765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.100494    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww" event={"ID":"c5a39c85-e039-4c53-896d-f81e4754917c","Type":"ContainerDied","Data":"9b47e6449d6a58080d921547dcc5fc4de1bceeb5c8c44cd7e39e24e7e45e87e0"}
Nov 02 01:29:30.100946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.100924    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e128ad-twpqk\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.101017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.100976    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e128ad-twpqk\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.102557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.102360    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz" event={"ID":"b0ae183c-f86b-4560-adca-9430c3f39eeb","Type":"ContainerDied","Data":"ed0ed057ed37fd518a2dbb46e621d2817ea812fdbf61cc2b269e480cdf6887c3"}
Nov 02 01:29:30.102557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.102395    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed0ed057ed37fd518a2dbb46e621d2817ea812fdbf61cc2b269e480cdf6887c3"
Nov 02 01:29:30.103052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.103034    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"
Nov 02 01:29:30.103701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.103681    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj" event={"ID":"c7a2d739-07d2-437e-bd57-806e0525ad24","Type":"ContainerStarted","Data":"6e384bd7606ab7ec4927d6ea0b4431595496c62238180bff83f0dec10a2490b0"}
Nov 02 01:29:30.105394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.105378    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7" event={"ID":"362c6a8e-907e-4aae-b272-412bbe0c848b","Type":"ContainerStarted","Data":"c5b7d4b86d78db9d52922e0c97d64852c75b2e56859fb04524b2672dc1693644"}
Nov 02 01:29:30.202268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.202237    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e128ad-twpqk\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.202395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.202300    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e128ad-twpqk\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.202644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.202619    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv762\" (UniqueName: \"kubernetes.io/projected/29356a60-2eeb-438b-a57b-a589b3faf0f8-kube-api-access-pv762\") pod \"maas-api-revoke-sub-b4e128ad-twpqk\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.203063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.203038    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e128ad-twpqk\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.205695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.205672    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e128ad-twpqk\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.216266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.216241    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv762\" (UniqueName: \"kubernetes.io/projected/29356a60-2eeb-438b-a57b-a589b3faf0f8-kube-api-access-pv762\") pod \"maas-api-revoke-sub-b4e128ad-twpqk\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.268366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.255487    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:30.329342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.324193    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:30.666317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.666219    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:29:30.666317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.666275    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 01:29:30.757254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.756808    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:29:30.757254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.756866    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 01:29:30.802057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:30.802017    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"]
Nov 02 01:29:30.817258 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:30.817230    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29356a60_2eeb_438b_a57b_a589b3faf0f8.slice/crio-d466880b59d69dc35aa887290ba90fd84dcfbfcc0795b2a756287838a6ed6fb0 WatchSource:0}: Error finding container d466880b59d69dc35aa887290ba90fd84dcfbfcc0795b2a756287838a6ed6fb0: Status 404 returned error can't find the container with id d466880b59d69dc35aa887290ba90fd84dcfbfcc0795b2a756287838a6ed6fb0
Nov 02 01:29:31.114807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.114712    4851 generic.go:356] "Generic (PLEG): container finished" podID="362c6a8e-907e-4aae-b272-412bbe0c848b" containerID="c5b7d4b86d78db9d52922e0c97d64852c75b2e56859fb04524b2672dc1693644" exitCode=0
Nov 02 01:29:31.114807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.114793    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7" event={"ID":"362c6a8e-907e-4aae-b272-412bbe0c848b","Type":"ContainerDied","Data":"c5b7d4b86d78db9d52922e0c97d64852c75b2e56859fb04524b2672dc1693644"}
Nov 02 01:29:31.116819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.116797    4851 generic.go:356] "Generic (PLEG): container finished" podID="c7a2d739-07d2-437e-bd57-806e0525ad24" containerID="36e168260dd6b80f8f1597aa2ea44b67ef28f371972afe46b3e46329b76cc229" exitCode=0
Nov 02 01:29:31.116944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.116922    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj" event={"ID":"c7a2d739-07d2-437e-bd57-806e0525ad24","Type":"ContainerDied","Data":"36e168260dd6b80f8f1597aa2ea44b67ef28f371972afe46b3e46329b76cc229"}
Nov 02 01:29:31.118357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.118333    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk" event={"ID":"29356a60-2eeb-438b-a57b-a589b3faf0f8","Type":"ContainerStarted","Data":"d466880b59d69dc35aa887290ba90fd84dcfbfcc0795b2a756287838a6ed6fb0"}
Nov 02 01:29:31.503923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.502861    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:31.688318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.688198    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:29:31.688318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.688253    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 01:29:31.701025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.701001    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:31.725964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.725922    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-openshift-ca" pod "362c6a8e-907e-4aae-b272-412bbe0c848b" (UID: "362c6a8e-907e-4aae-b272-412bbe0c848b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:31.726471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.726423    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-openshift-ca\") pod \"362c6a8e-907e-4aae-b272-412bbe0c848b\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") "
Nov 02 01:29:31.726584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.726539    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362c6a8e-907e-4aae-b272-412bbe0c848b-kube-api-access-pqxww\" (UniqueName: \"kubernetes.io/projected/362c6a8e-907e-4aae-b272-412bbe0c848b-kube-api-access-pqxww\") pod \"362c6a8e-907e-4aae-b272-412bbe0c848b\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") "
Nov 02 01:29:31.726659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.726644    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-xks-ca\") pod \"362c6a8e-907e-4aae-b272-412bbe0c848b\" (UID: \"362c6a8e-907e-4aae-b272-412bbe0c848b\") "
Nov 02 01:29:31.727154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.727135    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:31.729517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.729466    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:31.730133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.730112    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-xks-ca" pod "362c6a8e-907e-4aae-b272-412bbe0c848b" (UID: "362c6a8e-907e-4aae-b272-412bbe0c848b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:31.730219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.730162    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362c6a8e-907e-4aae-b272-412bbe0c848b-kube-api-access-pqxww" pod "362c6a8e-907e-4aae-b272-412bbe0c848b" (UID: "362c6a8e-907e-4aae-b272-412bbe0c848b"). InnerVolumeSpecName "kube-api-access-pqxww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:31.760314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.757218    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:29:31.760314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.757272    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 01:29:31.828530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.828498    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a39c85-e039-4c53-896d-f81e4754917c-kube-api-access-s95kh\" (UniqueName: \"kubernetes.io/projected/c5a39c85-e039-4c53-896d-f81e4754917c-kube-api-access-s95kh\") pod \"c5a39c85-e039-4c53-896d-f81e4754917c\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") "
Nov 02 01:29:31.828716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.828598    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-xks-ca\") pod \"c5a39c85-e039-4c53-896d-f81e4754917c\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") "
Nov 02 01:29:31.828812 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.828796    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-openshift-ca\") pod \"c5a39c85-e039-4c53-896d-f81e4754917c\" (UID: \"c5a39c85-e039-4c53-896d-f81e4754917c\") "
Nov 02 01:29:31.829365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.829342    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-openshift-ca" pod "c5a39c85-e039-4c53-896d-f81e4754917c" (UID: "c5a39c85-e039-4c53-896d-f81e4754917c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:31.829577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.829564    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqxww\" (UniqueName: \"kubernetes.io/projected/362c6a8e-907e-4aae-b272-412bbe0c848b-kube-api-access-pqxww\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:31.829663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.829652    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362c6a8e-907e-4aae-b272-412bbe0c848b-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:31.829739 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.829730    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:31.830801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.830786    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-xks-ca" pod "c5a39c85-e039-4c53-896d-f81e4754917c" (UID: "c5a39c85-e039-4c53-896d-f81e4754917c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:31.830942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.830922    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a39c85-e039-4c53-896d-f81e4754917c-kube-api-access-s95kh" pod "c5a39c85-e039-4c53-896d-f81e4754917c" (UID: "c5a39c85-e039-4c53-896d-f81e4754917c"). InnerVolumeSpecName "kube-api-access-s95kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:31.931317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.931122    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s95kh\" (UniqueName: \"kubernetes.io/projected/c5a39c85-e039-4c53-896d-f81e4754917c-kube-api-access-s95kh\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:31.931317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:31.931154    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a39c85-e039-4c53-896d-f81e4754917c-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:32.129221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.129194    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7" event={"ID":"362c6a8e-907e-4aae-b272-412bbe0c848b","Type":"ContainerDied","Data":"bb037ffca2857739874c4342dfcc88f70beea7e9f1ac7c6e563dd07e175f32b4"}
Nov 02 01:29:32.129418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.129406    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb037ffca2857739874c4342dfcc88f70beea7e9f1ac7c6e563dd07e175f32b4"
Nov 02 01:29:32.129501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.129221    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"
Nov 02 01:29:32.130535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.130518    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"
Nov 02 01:29:32.130945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.130559    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww" event={"ID":"c5a39c85-e039-4c53-896d-f81e4754917c","Type":"ContainerDied","Data":"fcbc50d0f56ef538a6b13b9cf9187d0e2bc5df9dd5aea665a8e5586a3e9f4f45"}
Nov 02 01:29:32.130945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.130784    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcbc50d0f56ef538a6b13b9cf9187d0e2bc5df9dd5aea665a8e5586a3e9f4f45"
Nov 02 01:29:32.144435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.144402    4851 generic.go:356] "Generic (PLEG): container finished" podID="29356a60-2eeb-438b-a57b-a589b3faf0f8" containerID="394daf5ab7d8b74bc3776e755dc130c7751b037aa6322590dfedaa504d3fc74f" exitCode=0
Nov 02 01:29:32.149610 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.149519    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk" event={"ID":"29356a60-2eeb-438b-a57b-a589b3faf0f8","Type":"ContainerDied","Data":"394daf5ab7d8b74bc3776e755dc130c7751b037aa6322590dfedaa504d3fc74f"}
Nov 02 01:29:32.286312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.284937    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:32.622711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.622688    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:32.688439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.678032    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:29:32.688439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.678088    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 01:29:32.707978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.707947    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-xks-ca\") pod \"c7a2d739-07d2-437e-bd57-806e0525ad24\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") "
Nov 02 01:29:32.708156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.707998    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a2d739-07d2-437e-bd57-806e0525ad24-kube-api-access-kk4bm\" (UniqueName: \"kubernetes.io/projected/c7a2d739-07d2-437e-bd57-806e0525ad24-kube-api-access-kk4bm\") pod \"c7a2d739-07d2-437e-bd57-806e0525ad24\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") "
Nov 02 01:29:32.708156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.708085    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-openshift-ca\") pod \"c7a2d739-07d2-437e-bd57-806e0525ad24\" (UID: \"c7a2d739-07d2-437e-bd57-806e0525ad24\") "
Nov 02 01:29:32.708476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.708445    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-openshift-ca" pod "c7a2d739-07d2-437e-bd57-806e0525ad24" (UID: "c7a2d739-07d2-437e-bd57-806e0525ad24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:32.708782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.708753    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:32.710328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.710264    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a2d739-07d2-437e-bd57-806e0525ad24-kube-api-access-kk4bm" pod "c7a2d739-07d2-437e-bd57-806e0525ad24" (UID: "c7a2d739-07d2-437e-bd57-806e0525ad24"). InnerVolumeSpecName "kube-api-access-kk4bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:32.710527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.710512    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-xks-ca" pod "c7a2d739-07d2-437e-bd57-806e0525ad24" (UID: "c7a2d739-07d2-437e-bd57-806e0525ad24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:32.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.758668    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:29:32.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.758733    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 01:29:32.810991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.810961    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a2d739-07d2-437e-bd57-806e0525ad24-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:32.810991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:32.810989    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk4bm\" (UniqueName: \"kubernetes.io/projected/c7a2d739-07d2-437e-bd57-806e0525ad24-kube-api-access-kk4bm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:33.152394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.142903    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:33.226313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.221899    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"
Nov 02 01:29:33.253317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.229615    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj" event={"ID":"c7a2d739-07d2-437e-bd57-806e0525ad24","Type":"ContainerDied","Data":"6e384bd7606ab7ec4927d6ea0b4431595496c62238180bff83f0dec10a2490b0"}
Nov 02 01:29:33.253317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.229700    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e384bd7606ab7ec4927d6ea0b4431595496c62238180bff83f0dec10a2490b0"
Nov 02 01:29:33.622106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.622083    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:33.669368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.663877    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:29:33.669368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.663930    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 01:29:33.730529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.730493    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-openshift-ca\") pod \"29356a60-2eeb-438b-a57b-a589b3faf0f8\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") "
Nov 02 01:29:33.730915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.730601    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29356a60-2eeb-438b-a57b-a589b3faf0f8-kube-api-access-pv762\" (UniqueName: \"kubernetes.io/projected/29356a60-2eeb-438b-a57b-a589b3faf0f8-kube-api-access-pv762\") pod \"29356a60-2eeb-438b-a57b-a589b3faf0f8\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") "
Nov 02 01:29:33.730915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.730719    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-xks-ca\") pod \"29356a60-2eeb-438b-a57b-a589b3faf0f8\" (UID: \"29356a60-2eeb-438b-a57b-a589b3faf0f8\") "
Nov 02 01:29:33.731233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.731205    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-openshift-ca" pod "29356a60-2eeb-438b-a57b-a589b3faf0f8" (UID: "29356a60-2eeb-438b-a57b-a589b3faf0f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:33.731408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.731392    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:33.762808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.755611    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:29:33.762808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.755666    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 01:29:33.762808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.756169    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29356a60-2eeb-438b-a57b-a589b3faf0f8-kube-api-access-pv762" pod "29356a60-2eeb-438b-a57b-a589b3faf0f8" (UID: "29356a60-2eeb-438b-a57b-a589b3faf0f8"). InnerVolumeSpecName "kube-api-access-pv762". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:33.794265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.793457    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-xks-ca" pod "29356a60-2eeb-438b-a57b-a589b3faf0f8" (UID: "29356a60-2eeb-438b-a57b-a589b3faf0f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:33.879312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.857586    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv762\" (UniqueName: \"kubernetes.io/projected/29356a60-2eeb-438b-a57b-a589b3faf0f8-kube-api-access-pv762\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:33.879312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.857623    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29356a60-2eeb-438b-a57b-a589b3faf0f8-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:33.894363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:33.893918    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:34.117389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.117304    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"]
Nov 02 01:29:34.117989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.117967    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c5a39c85-e039-4c53-896d-f81e4754917c" containerName="revoke-keys"
Nov 02 01:29:34.118089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.117991    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a39c85-e039-4c53-896d-f81e4754917c" containerName="revoke-keys"
Nov 02 01:29:34.118089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.118030    4851 cpu_manager.go:414] "containerMap: removing container" podUID="29356a60-2eeb-438b-a57b-a589b3faf0f8" containerName="revoke-keys"
Nov 02 01:29:34.118089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.118038    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29356a60-2eeb-438b-a57b-a589b3faf0f8" containerName="revoke-keys"
Nov 02 01:29:34.118089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.118083    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c7a2d739-07d2-437e-bd57-806e0525ad24" containerName="revoke-keys"
Nov 02 01:29:34.118308 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.118092    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a2d739-07d2-437e-bd57-806e0525ad24" containerName="revoke-keys"
Nov 02 01:29:34.118308 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.118102    4851 cpu_manager.go:414] "containerMap: removing container" podUID="362c6a8e-907e-4aae-b272-412bbe0c848b" containerName="revoke-keys"
Nov 02 01:29:34.118308 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.118112    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362c6a8e-907e-4aae-b272-412bbe0c848b" containerName="revoke-keys"
Nov 02 01:29:34.118308 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.118279    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="29356a60-2eeb-438b-a57b-a589b3faf0f8" containerName="revoke-keys"
Nov 02 01:29:34.118505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.118319    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a2d739-07d2-437e-bd57-806e0525ad24" containerName="revoke-keys"
Nov 02 01:29:34.118505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.118330    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="362c6a8e-907e-4aae-b272-412bbe0c848b" containerName="revoke-keys"
Nov 02 01:29:34.118505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.118346    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a39c85-e039-4c53-896d-f81e4754917c" containerName="revoke-keys"
Nov 02 01:29:34.122242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.122221    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.124507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.124482    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"]
Nov 02 01:29:34.231126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.231096    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk" event={"ID":"29356a60-2eeb-438b-a57b-a589b3faf0f8","Type":"ContainerDied","Data":"d466880b59d69dc35aa887290ba90fd84dcfbfcc0795b2a756287838a6ed6fb0"}
Nov 02 01:29:34.231304 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.231140    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d466880b59d69dc35aa887290ba90fd84dcfbfcc0795b2a756287838a6ed6fb0"
Nov 02 01:29:34.231304 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.231110    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"
Nov 02 01:29:34.290705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.290675    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba0e38cb-zks4t\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.290915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.290898    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fspb\" (UniqueName: \"kubernetes.io/projected/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-kube-api-access-8fspb\") pod \"maas-api-revoke-sub-ba0e38cb-zks4t\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.290965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.290951    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba0e38cb-zks4t\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.365323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.364743    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:34.391922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.391840    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fspb\" (UniqueName: \"kubernetes.io/projected/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-kube-api-access-8fspb\") pod \"maas-api-revoke-sub-ba0e38cb-zks4t\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.391922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.391901    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba0e38cb-zks4t\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.392103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.392023    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba0e38cb-zks4t\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.392661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.392628    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba0e38cb-zks4t\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.394534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.394509    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba0e38cb-zks4t\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.404567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.404545    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fspb\" (UniqueName: \"kubernetes.io/projected/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-kube-api-access-8fspb\") pod \"maas-api-revoke-sub-ba0e38cb-zks4t\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.494402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.494353    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"]
Nov 02 01:29:34.498692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.498650    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:34.512897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.512812    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:34.516716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.516657    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"]
Nov 02 01:29:34.516792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.516759    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.601200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.601171    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d4d948-qzqt8\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.601479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.601462    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbc4v\" (UniqueName: \"kubernetes.io/projected/a8d77d12-7108-4358-91f7-08057e46435c-kube-api-access-qbc4v\") pod \"maas-api-revoke-sub-60d4d948-qzqt8\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.601616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.601579    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d4d948-qzqt8\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.648894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.647788    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:29:34.648894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.647849    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 01:29:34.702967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.702915    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbc4v\" (UniqueName: \"kubernetes.io/projected/a8d77d12-7108-4358-91f7-08057e46435c-kube-api-access-qbc4v\") pod \"maas-api-revoke-sub-60d4d948-qzqt8\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.703116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.702981    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d4d948-qzqt8\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.703182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.703108    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d4d948-qzqt8\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.703732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.703712    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d4d948-qzqt8\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.705880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.705859    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d4d948-qzqt8\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.715860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.715831    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbc4v\" (UniqueName: \"kubernetes.io/projected/a8d77d12-7108-4358-91f7-08057e46435c-kube-api-access-qbc4v\") pod \"maas-api-revoke-sub-60d4d948-qzqt8\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.743816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.743787    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:29:34.744173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.743859    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 01:29:34.820538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.820507    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:34.928414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.906623    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:34.989014 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:34.988977    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"]
Nov 02 01:29:35.243498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:35.243415    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t" event={"ID":"eacc2ba5-4848-44bb-8253-e4e0ae9c1300","Type":"ContainerStarted","Data":"770da33f4d42d8dc7213b66455bb467520f7b972d32b1ca82ceec500d261e790"}
Nov 02 01:29:35.282545 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:35.282508    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d77d12_7108_4358_91f7_08057e46435c.slice/crio-6a105ff56621b238ba224fe6816ac24e12c0a0448aed55358eb4553d1cf44654 WatchSource:0}: Error finding container 6a105ff56621b238ba224fe6816ac24e12c0a0448aed55358eb4553d1cf44654: Status 404 returned error can't find the container with id 6a105ff56621b238ba224fe6816ac24e12c0a0448aed55358eb4553d1cf44654
Nov 02 01:29:35.285591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:35.285551    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"]
Nov 02 01:29:35.613440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:35.613408    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:35.653344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:35.652758    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:29:35.653344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:35.652813    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 01:29:36.253368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.253335    4851 generic.go:356] "Generic (PLEG): container finished" podID="a8d77d12-7108-4358-91f7-08057e46435c" containerID="b857e1148f9e0dd33ebbead7ab4097ea06e25e7bb208137536725cbf105dc0e4" exitCode=0
Nov 02 01:29:36.253810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.253420    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8" event={"ID":"a8d77d12-7108-4358-91f7-08057e46435c","Type":"ContainerDied","Data":"b857e1148f9e0dd33ebbead7ab4097ea06e25e7bb208137536725cbf105dc0e4"}
Nov 02 01:29:36.253810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.253475    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8" event={"ID":"a8d77d12-7108-4358-91f7-08057e46435c","Type":"ContainerStarted","Data":"6a105ff56621b238ba224fe6816ac24e12c0a0448aed55358eb4553d1cf44654"}
Nov 02 01:29:36.255218 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.255188    4851 generic.go:356] "Generic (PLEG): container finished" podID="eacc2ba5-4848-44bb-8253-e4e0ae9c1300" containerID="77b848b15aec3569b988502ab8ee29aea247569c6415e70fd8d3d55046c1407a" exitCode=0
Nov 02 01:29:36.255372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.255277    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t" event={"ID":"eacc2ba5-4848-44bb-8253-e4e0ae9c1300","Type":"ContainerDied","Data":"77b848b15aec3569b988502ab8ee29aea247569c6415e70fd8d3d55046c1407a"}
Nov 02 01:29:36.305117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.305049    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"]
Nov 02 01:29:36.334334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.329568    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.338236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.338211    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"]
Nov 02 01:29:36.346367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.345930    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-890c7d8b-x2s2k\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.346367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.346106    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl9gc\" (UniqueName: \"kubernetes.io/projected/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-kube-api-access-dl9gc\") pod \"maas-api-revoke-sub-890c7d8b-x2s2k\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.346367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.346160    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-890c7d8b-x2s2k\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.385779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.385749    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:36.386821 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.386797    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:36.387681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.387658    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:29:36.447693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.447657    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-890c7d8b-x2s2k\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.447874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.447798    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl9gc\" (UniqueName: \"kubernetes.io/projected/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-kube-api-access-dl9gc\") pod \"maas-api-revoke-sub-890c7d8b-x2s2k\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.447938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.447884    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-890c7d8b-x2s2k\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.449016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.448986    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-890c7d8b-x2s2k\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.454646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.454617    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-890c7d8b-x2s2k\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.455234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.455212    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:36.462041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.462021    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl9gc\" (UniqueName: \"kubernetes.io/projected/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-kube-api-access-dl9gc\") pod \"maas-api-revoke-sub-890c7d8b-x2s2k\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.651541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.651471    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:36.651944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.651928    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:36.652639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.652620    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:29:36.674415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.674381    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:36.977992 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:36.977968    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3ed0ec_2347_4765_982b_28e52a7d1fd1.slice/crio-60d21e32f9aa23fb11303d736a882c9d154c45bfe81a19724b79084fab414ec2 WatchSource:0}: Error finding container 60d21e32f9aa23fb11303d736a882c9d154c45bfe81a19724b79084fab414ec2: Status 404 returned error can't find the container with id 60d21e32f9aa23fb11303d736a882c9d154c45bfe81a19724b79084fab414ec2
Nov 02 01:29:36.979580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:36.979557    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"]
Nov 02 01:29:37.130761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.130717    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:37.265315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.265204    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k" event={"ID":"eb3ed0ec-2347-4765-982b-28e52a7d1fd1","Type":"ContainerStarted","Data":"60d21e32f9aa23fb11303d736a882c9d154c45bfe81a19724b79084fab414ec2"}
Nov 02 01:29:37.478489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.478460    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:37.481857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.481838    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:37.570751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.570664    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-xks-ca\") pod \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") "
Nov 02 01:29:37.570925 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.570905    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-kube-api-access-8fspb\" (UniqueName: \"kubernetes.io/projected/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-kube-api-access-8fspb\") pod \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") "
Nov 02 01:29:37.571016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.570994    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d77d12-7108-4358-91f7-08057e46435c-kube-api-access-qbc4v\" (UniqueName: \"kubernetes.io/projected/a8d77d12-7108-4358-91f7-08057e46435c-kube-api-access-qbc4v\") pod \"a8d77d12-7108-4358-91f7-08057e46435c\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") "
Nov 02 01:29:37.571123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.571036    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-openshift-ca\") pod \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\" (UID: \"eacc2ba5-4848-44bb-8253-e4e0ae9c1300\") "
Nov 02 01:29:37.571123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.571055    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-openshift-ca\") pod \"a8d77d12-7108-4358-91f7-08057e46435c\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") "
Nov 02 01:29:37.571123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.571087    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-xks-ca\") pod \"a8d77d12-7108-4358-91f7-08057e46435c\" (UID: \"a8d77d12-7108-4358-91f7-08057e46435c\") "
Nov 02 01:29:37.571467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.571448    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-openshift-ca" pod "eacc2ba5-4848-44bb-8253-e4e0ae9c1300" (UID: "eacc2ba5-4848-44bb-8253-e4e0ae9c1300"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:37.571694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.571668    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-openshift-ca" pod "a8d77d12-7108-4358-91f7-08057e46435c" (UID: "a8d77d12-7108-4358-91f7-08057e46435c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:37.571915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.571893    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:37.572972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.572950    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-xks-ca" pod "eacc2ba5-4848-44bb-8253-e4e0ae9c1300" (UID: "eacc2ba5-4848-44bb-8253-e4e0ae9c1300"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:37.573174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.573155    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-kube-api-access-8fspb" pod "eacc2ba5-4848-44bb-8253-e4e0ae9c1300" (UID: "eacc2ba5-4848-44bb-8253-e4e0ae9c1300"). InnerVolumeSpecName "kube-api-access-8fspb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:37.573258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.573239    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d77d12-7108-4358-91f7-08057e46435c-kube-api-access-qbc4v" pod "a8d77d12-7108-4358-91f7-08057e46435c" (UID: "a8d77d12-7108-4358-91f7-08057e46435c"). InnerVolumeSpecName "kube-api-access-qbc4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:37.573493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.573477    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-xks-ca" pod "a8d77d12-7108-4358-91f7-08057e46435c" (UID: "a8d77d12-7108-4358-91f7-08057e46435c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:37.673161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.673119    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fspb\" (UniqueName: \"kubernetes.io/projected/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-kube-api-access-8fspb\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:37.673161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.673152    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbc4v\" (UniqueName: \"kubernetes.io/projected/a8d77d12-7108-4358-91f7-08057e46435c-kube-api-access-qbc4v\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:37.673161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.673166    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:37.673387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.673180    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d77d12-7108-4358-91f7-08057e46435c-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:37.673387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.673194    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eacc2ba5-4848-44bb-8253-e4e0ae9c1300-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:37.825956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:37.825876    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:38.276005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.275976    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"
Nov 02 01:29:38.276497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.276004    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8" event={"ID":"a8d77d12-7108-4358-91f7-08057e46435c","Type":"ContainerDied","Data":"6a105ff56621b238ba224fe6816ac24e12c0a0448aed55358eb4553d1cf44654"}
Nov 02 01:29:38.276497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.276057    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a105ff56621b238ba224fe6816ac24e12c0a0448aed55358eb4553d1cf44654"
Nov 02 01:29:38.277643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.277612    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"
Nov 02 01:29:38.277798 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.277616    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t" event={"ID":"eacc2ba5-4848-44bb-8253-e4e0ae9c1300","Type":"ContainerDied","Data":"770da33f4d42d8dc7213b66455bb467520f7b972d32b1ca82ceec500d261e790"}
Nov 02 01:29:38.277798 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.277723    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="770da33f4d42d8dc7213b66455bb467520f7b972d32b1ca82ceec500d261e790"
Nov 02 01:29:38.279547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.279510    4851 generic.go:356] "Generic (PLEG): container finished" podID="eb3ed0ec-2347-4765-982b-28e52a7d1fd1" containerID="b598f8970b2af86572800e22efc3db592b9901b84813b6f585bba6d6f5c1ee52" exitCode=0
Nov 02 01:29:38.279652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.279599    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k" event={"ID":"eb3ed0ec-2347-4765-982b-28e52a7d1fd1","Type":"ContainerDied","Data":"b598f8970b2af86572800e22efc3db592b9901b84813b6f585bba6d6f5c1ee52"}
Nov 02 01:29:38.356634 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.356605    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"]
Nov 02 01:29:38.357237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.357222    4851 cpu_manager.go:414] "containerMap: removing container" podUID="eacc2ba5-4848-44bb-8253-e4e0ae9c1300" containerName="revoke-keys"
Nov 02 01:29:38.357275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.357239    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eacc2ba5-4848-44bb-8253-e4e0ae9c1300" containerName="revoke-keys"
Nov 02 01:29:38.357339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.357274    4851 cpu_manager.go:414] "containerMap: removing container" podUID="a8d77d12-7108-4358-91f7-08057e46435c" containerName="revoke-keys"
Nov 02 01:29:38.357339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.357280    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d77d12-7108-4358-91f7-08057e46435c" containerName="revoke-keys"
Nov 02 01:29:38.357400 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.357391    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d77d12-7108-4358-91f7-08057e46435c" containerName="revoke-keys"
Nov 02 01:29:38.357434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.357404    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="eacc2ba5-4848-44bb-8253-e4e0ae9c1300" containerName="revoke-keys"
Nov 02 01:29:38.361087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.361072    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.386709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.386679    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"]
Nov 02 01:29:38.388266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.388227    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84d6l\" (UniqueName: \"kubernetes.io/projected/763c53db-cc32-40fe-a758-67b9d7ccbef9-kube-api-access-84d6l\") pod \"maas-api-revoke-sub-07ff9fb7-4r9rn\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.388508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.388483    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07ff9fb7-4r9rn\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.388648 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.388552    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07ff9fb7-4r9rn\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.490374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.490334    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d6l\" (UniqueName: \"kubernetes.io/projected/763c53db-cc32-40fe-a758-67b9d7ccbef9-kube-api-access-84d6l\") pod \"maas-api-revoke-sub-07ff9fb7-4r9rn\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.490551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.490494    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07ff9fb7-4r9rn\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.490600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.490549    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07ff9fb7-4r9rn\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.491243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.491223    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07ff9fb7-4r9rn\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.493008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.492987    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07ff9fb7-4r9rn\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.502513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.502488    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84d6l\" (UniqueName: \"kubernetes.io/projected/763c53db-cc32-40fe-a758-67b9d7ccbef9-kube-api-access-84d6l\") pod \"maas-api-revoke-sub-07ff9fb7-4r9rn\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.681016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.680986    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:38.777185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.776927    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:38.966439 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:38.966385    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763c53db_cc32_40fe_a758_67b9d7ccbef9.slice/crio-aafaba06b59e83e5f7aee64241a058d57956ec32e14d9be620a6a944f4bc57e7 WatchSource:0}: Error finding container aafaba06b59e83e5f7aee64241a058d57956ec32e14d9be620a6a944f4bc57e7: Status 404 returned error can't find the container with id aafaba06b59e83e5f7aee64241a058d57956ec32e14d9be620a6a944f4bc57e7
Nov 02 01:29:38.969406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:38.969382    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"]
Nov 02 01:29:39.096923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.096893    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"]
Nov 02 01:29:39.105400 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.104588    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.116504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.116473    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.116639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.116575    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbm87\" (UniqueName: \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-kube-api-access-wbm87\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.116639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.116619    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.116762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.116661    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.116762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.116726    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.116858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.116807    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.116858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.116832    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.116962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.116889    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.117016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.116975    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.121551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.121527    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"]
Nov 02 01:29:39.217909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.217845    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbm87\" (UniqueName: \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-kube-api-access-wbm87\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.217909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.217888    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.217919    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.217958    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.218097    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.218134    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.218190    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.218275    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.218375    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.218482    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218564 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.218519    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.218566    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.218688    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.218981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.218959    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.220438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.220418    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.220927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.220912    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.229945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.229918    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.230839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.230818    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbm87\" (UniqueName: \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-kube-api-access-wbm87\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.288943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.288899    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn" event={"ID":"763c53db-cc32-40fe-a758-67b9d7ccbef9","Type":"ContainerStarted","Data":"9117e5f045bfe09ce52ac871e2ccf47f7d069e80079e894da4965f4f33893069"}
Nov 02 01:29:39.289381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.288964    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn" event={"ID":"763c53db-cc32-40fe-a758-67b9d7ccbef9","Type":"ContainerStarted","Data":"aafaba06b59e83e5f7aee64241a058d57956ec32e14d9be620a6a944f4bc57e7"}
Nov 02 01:29:39.318484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.318427    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn" podStartSLOduration=1.318414138 podStartE2EDuration="1.318414138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:39.311531184 +0000 UTC m=+884.616464011" watchObservedRunningTime="2026-11-02 01:29:39.318414138 +0000 UTC m=+884.623346980"
Nov 02 01:29:39.428395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.428370    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:39.437581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.437552    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:39.501142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.501113    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"]
Nov 02 01:29:39.502189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.502167    4851 cpu_manager.go:414] "containerMap: removing container" podUID="eb3ed0ec-2347-4765-982b-28e52a7d1fd1" containerName="revoke-keys"
Nov 02 01:29:39.502246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.502192    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb3ed0ec-2347-4765-982b-28e52a7d1fd1" containerName="revoke-keys"
Nov 02 01:29:39.502378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.502365    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb3ed0ec-2347-4765-982b-28e52a7d1fd1" containerName="revoke-keys"
Nov 02 01:29:39.507492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.507206    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.519857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.519833    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"]
Nov 02 01:29:39.522698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.522572    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.522698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.522642    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.522698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.522675    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d43d33e-3b63-4c69-ae89-480cef886400-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.522904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.522700    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.522904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.522733    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.522904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.522794    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.523059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.522943    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.523112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.523072    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5nkp\" (UniqueName: \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-kube-api-access-c5nkp\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.523181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.523164    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d43d33e-3b63-4c69-ae89-480cef886400-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.526012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.525989    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.526124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.526041    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"]
Nov 02 01:29:39.532088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.532063    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"]
Nov 02 01:29:39.554241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.554222    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"]
Nov 02 01:29:39.554468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.554450    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.559647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.559623    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"]
Nov 02 01:29:39.567935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.567906    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:39.635308 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.635245    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-kube-api-access-dl9gc\" (UniqueName: \"kubernetes.io/projected/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-kube-api-access-dl9gc\") pod \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") "
Nov 02 01:29:39.635506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.635391    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-openshift-ca\") pod \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") "
Nov 02 01:29:39.638363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.638333    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-xks-ca\") pod \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\" (UID: \"eb3ed0ec-2347-4765-982b-28e52a7d1fd1\") "
Nov 02 01:29:39.639365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.639245    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.639517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.639466    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.639643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.639627    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.639807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.639782    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5nkp\" (UniqueName: \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-kube-api-access-c5nkp\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.639921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.639905    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d43d33e-3b63-4c69-ae89-480cef886400-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.639991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.639976    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.640223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.640053    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.640314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.640271    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d43d33e-3b63-4c69-ae89-480cef886400-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.640367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.640340    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.644656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.644622    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-openshift-ca" pod "eb3ed0ec-2347-4765-982b-28e52a7d1fd1" (UID: "eb3ed0ec-2347-4765-982b-28e52a7d1fd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:39.648579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.648530    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.672313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.669020    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.672313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.669322    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.672313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.671148    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.672313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.672092    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d43d33e-3b63-4c69-ae89-480cef886400-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.686313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.679549    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5nkp\" (UniqueName: \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-kube-api-access-c5nkp\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.686313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.683935    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d43d33e-3b63-4c69-ae89-480cef886400-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.693998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.693907    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-xks-ca" pod "eb3ed0ec-2347-4765-982b-28e52a7d1fd1" (UID: "eb3ed0ec-2347-4765-982b-28e52a7d1fd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:39.712595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.712520    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-kube-api-access-dl9gc" pod "eb3ed0ec-2347-4765-982b-28e52a7d1fd1" (UID: "eb3ed0ec-2347-4765-982b-28e52a7d1fd1"). InnerVolumeSpecName "kube-api-access-dl9gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:39.723099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.723038    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.723466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.723444    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.741881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.741844    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.742017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.741898    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.742017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.741949    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.742017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.741991    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.742120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742016    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgrv\" (UniqueName: \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-kube-api-access-khgrv\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.742120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742043    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.742120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742073    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c0169e-44d5-4070-83f8-fbf617820e95-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.742312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742128    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.742312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742179    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c0169e-44d5-4070-83f8-fbf617820e95-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.742312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742226    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.742312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742271    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.742521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742398    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.742521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742471    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd465d06-c445-4b09-a15f-5d97797c850c-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.742620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742540    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.742620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742562    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8twd\" (UniqueName: \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-kube-api-access-p8twd\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.742620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742607    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.742760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742644    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.742760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742691    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd465d06-c445-4b09-a15f-5d97797c850c-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.742867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742792    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:39.742867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742809    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl9gc\" (UniqueName: \"kubernetes.io/projected/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-kube-api-access-dl9gc\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:39.742867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.742825    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3ed0ec-2347-4765-982b-28e52a7d1fd1-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:39.833545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.833511    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:39.844092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844054    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.844277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844255    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.844476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844451    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844508    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844536    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgrv\" (UniqueName: \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-kube-api-access-khgrv\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844565    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844594    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c0169e-44d5-4070-83f8-fbf617820e95-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844606    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844646    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844704    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c0169e-44d5-4070-83f8-fbf617820e95-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844754    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844801    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844912    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.844942    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd465d06-c445-4b09-a15f-5d97797c850c-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.845001    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.845031    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8twd\" (UniqueName: \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-kube-api-access-p8twd\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.845075    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.845111    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.845398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.845375    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.846121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.845607    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.846121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.846070    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.846966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.846479    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.847372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.847352    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.847758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.847737    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.847939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.847914    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd465d06-c445-4b09-a15f-5d97797c850c-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.848655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.848626    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.848946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.848925    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd465d06-c445-4b09-a15f-5d97797c850c-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.849265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.849238    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c0169e-44d5-4070-83f8-fbf617820e95-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.850205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.849507    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.850347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.849774    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd465d06-c445-4b09-a15f-5d97797c850c-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.850424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.850403    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.854819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.854798    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c0169e-44d5-4070-83f8-fbf617820e95-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.857767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.857688    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgrv\" (UniqueName: \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-kube-api-access-khgrv\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.859864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.859703    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:39.860215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.860195    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.862594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.862571    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8twd\" (UniqueName: \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-kube-api-access-p8twd\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.916410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.916380    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:39.936642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.936614    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"]
Nov 02 01:29:39.976131 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:39.975956    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8d238c_56bd_46bc_b233_c1c0bd398cfe.slice/crio-466d3fab508890f2094c2dbce83f2675c7c5841021672a78ce2d57cea1d156e4 WatchSource:0}: Error finding container 466d3fab508890f2094c2dbce83f2675c7c5841021672a78ce2d57cea1d156e4: Status 404 returned error can't find the container with id 466d3fab508890f2094c2dbce83f2675c7c5841021672a78ce2d57cea1d156e4
Nov 02 01:29:39.979106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.979068    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:39.979200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.979161    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:39.979236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:39.979207    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:40.151015 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:40.149900    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d43d33e_3b63_4c69_ae89_480cef886400.slice/crio-411d5f9853168075b3c46e20be07510a81a228490f4e3462c5548e572498774a WatchSource:0}: Error finding container 411d5f9853168075b3c46e20be07510a81a228490f4e3462c5548e572498774a: Status 404 returned error can't find the container with id 411d5f9853168075b3c46e20be07510a81a228490f4e3462c5548e572498774a
Nov 02 01:29:40.151927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.151651    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:40.153768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.153730    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:40.153897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.153811    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:40.153897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.153819    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"]
Nov 02 01:29:40.153897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.153852    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:40.281361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.280717    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"]
Nov 02 01:29:40.288542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.288508    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:40.289171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.289146    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:40.289714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.289689    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:40.330188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.330153    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" event={"ID":"dd465d06-c445-4b09-a15f-5d97797c850c","Type":"ContainerStarted","Data":"71750d215cd0fff425a8a6cd63f9edaf81d0a57e023d817998a60eb0970008e4"}
Nov 02 01:29:40.333254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.332142    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" event={"ID":"0a8d238c-56bd-46bc-b233-c1c0bd398cfe","Type":"ContainerStarted","Data":"e6e832bda8cd39fac1bb903fff56a2ce4275d404ae25c966249d8b4297db960b"}
Nov 02 01:29:40.333254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.332193    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" event={"ID":"0a8d238c-56bd-46bc-b233-c1c0bd398cfe","Type":"ContainerStarted","Data":"466d3fab508890f2094c2dbce83f2675c7c5841021672a78ce2d57cea1d156e4"}
Nov 02 01:29:40.338391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.335274    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" event={"ID":"3d43d33e-3b63-4c69-ae89-480cef886400","Type":"ContainerStarted","Data":"ad6a3825f1f3291edcdd94abfd5bf160d89757355ae8b500e374e08c80c37d69"}
Nov 02 01:29:40.338391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.337131    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" event={"ID":"3d43d33e-3b63-4c69-ae89-480cef886400","Type":"ContainerStarted","Data":"411d5f9853168075b3c46e20be07510a81a228490f4e3462c5548e572498774a"}
Nov 02 01:29:40.350334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.350162    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k" event={"ID":"eb3ed0ec-2347-4765-982b-28e52a7d1fd1","Type":"ContainerDied","Data":"60d21e32f9aa23fb11303d736a882c9d154c45bfe81a19724b79084fab414ec2"}
Nov 02 01:29:40.350334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.350208    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60d21e32f9aa23fb11303d736a882c9d154c45bfe81a19724b79084fab414ec2"
Nov 02 01:29:40.350893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.350281    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"
Nov 02 01:29:40.357126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.357103    4851 generic.go:356] "Generic (PLEG): container finished" podID="763c53db-cc32-40fe-a758-67b9d7ccbef9" containerID="9117e5f045bfe09ce52ac871e2ccf47f7d069e80079e894da4965f4f33893069" exitCode=0
Nov 02 01:29:40.357338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.357323    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn" event={"ID":"763c53db-cc32-40fe-a758-67b9d7ccbef9","Type":"ContainerDied","Data":"9117e5f045bfe09ce52ac871e2ccf47f7d069e80079e894da4965f4f33893069"}
Nov 02 01:29:40.383079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.383024    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podStartSLOduration=1.38300545 podStartE2EDuration="1.38300545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:40.370185635 +0000 UTC m=+885.675118463" watchObservedRunningTime="2026-11-02 01:29:40.38300545 +0000 UTC m=+885.687938281"
Nov 02 01:29:40.429108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.428811    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:40.446521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.441629    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podStartSLOduration=1.4416077 podStartE2EDuration="1.4416077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:40.402727271 +0000 UTC m=+885.707660100" watchObservedRunningTime="2026-11-02 01:29:40.4416077 +0000 UTC m=+885.746540527"
Nov 02 01:29:40.511339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.510828    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"]
Nov 02 01:29:40.513425 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:40.513386    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c0169e_44d5_4070_83f8_fbf617820e95.slice/crio-2498c180159843fe0a5ef9583f267bd7779cade46a43bf2a7c7574459e6c3624 WatchSource:0}: Error finding container 2498c180159843fe0a5ef9583f267bd7779cade46a43bf2a7c7574459e6c3624: Status 404 returned error can't find the container with id 2498c180159843fe0a5ef9583f267bd7779cade46a43bf2a7c7574459e6c3624
Nov 02 01:29:40.518266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.517604    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:40.518266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.517666    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:40.518266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.517698    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:40.694314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.688924    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:40.838503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:40.838373    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:41.422006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:41.421970    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" event={"ID":"dd465d06-c445-4b09-a15f-5d97797c850c","Type":"ContainerStarted","Data":"1ca66c6a16d173d4844902705250627ccf6eb79c453d43f0bf651bd750ef7ab0"}
Nov 02 01:29:41.427814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:41.426759    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" event={"ID":"39c0169e-44d5-4070-83f8-fbf617820e95","Type":"ContainerStarted","Data":"c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9"}
Nov 02 01:29:41.427814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:41.426809    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" event={"ID":"39c0169e-44d5-4070-83f8-fbf617820e95","Type":"ContainerStarted","Data":"2498c180159843fe0a5ef9583f267bd7779cade46a43bf2a7c7574459e6c3624"}
Nov 02 01:29:41.431686 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:41.431658    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:29:41.431881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:41.431861    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 01:29:41.480332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:41.480241    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podStartSLOduration=2.480217789 podStartE2EDuration="2.480217789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:41.457952975 +0000 UTC m=+886.762885813" watchObservedRunningTime="2026-11-02 01:29:41.480217789 +0000 UTC m=+886.785150616"
Nov 02 01:29:41.594315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:41.587829    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podStartSLOduration=2.587806342 podStartE2EDuration="2.587806342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:41.51105652 +0000 UTC m=+886.815989346" watchObservedRunningTime="2026-11-02 01:29:41.587806342 +0000 UTC m=+886.892739168"
Nov 02 01:29:41.853370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:41.844970    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:41.868957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:41.865433    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:29:41.868957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:41.865494    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 01:29:42.228347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.216264    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:42.269073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.269041    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:42.416977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.416936    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 01:29:42.417258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.417229    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 01:29:42.493424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.466067    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:42.516254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.508350    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:29:42.516254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.508412    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 01:29:42.519374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.519087    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:29:42.519374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.519145    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:29:42.688319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.661973    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:42.705911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.690975    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 02 01:29:42.794422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.794264    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:42.831625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.816635    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:42.863184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.828934    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:42.890620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.890568    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:42.902471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.899344    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:29:42.902471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.899427    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 01:29:42.970686 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.957005    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:29:42.985314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.971754    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded"
Nov 02 01:29:42.985642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.957044    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:42.985862 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.957010    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:29:42.986064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:42.982058    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:29:43.132721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.132610    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:43.168939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.146511    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 01:29:43.169260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.169223    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 01:29:43.195316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.187687    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:29:43.195316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.187763    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:29:43.225500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.225469    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:43.283316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.269679    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:43.283316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.269787    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:43.582634 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.582586    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:29:43.618599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.586698    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-openshift-ca\") pod \"763c53db-cc32-40fe-a758-67b9d7ccbef9\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") "
Nov 02 01:29:43.645431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.645402    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763c53db-cc32-40fe-a758-67b9d7ccbef9-kube-api-access-84d6l\" (UniqueName: \"kubernetes.io/projected/763c53db-cc32-40fe-a758-67b9d7ccbef9-kube-api-access-84d6l\") pod \"763c53db-cc32-40fe-a758-67b9d7ccbef9\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") "
Nov 02 01:29:43.647406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.590631    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:29:43.647406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.646170    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 01:29:43.647406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.589424    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:29:43.647406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.646209    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-xks-ca\") pod \"763c53db-cc32-40fe-a758-67b9d7ccbef9\" (UID: \"763c53db-cc32-40fe-a758-67b9d7ccbef9\") "
Nov 02 01:29:43.647406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.618892    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:43.647406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.645252    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-openshift-ca" pod "763c53db-cc32-40fe-a758-67b9d7ccbef9" (UID: "763c53db-cc32-40fe-a758-67b9d7ccbef9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:43.650795 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.650768    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763c53db-cc32-40fe-a758-67b9d7ccbef9-kube-api-access-84d6l" pod "763c53db-cc32-40fe-a758-67b9d7ccbef9" (UID: "763c53db-cc32-40fe-a758-67b9d7ccbef9"). InnerVolumeSpecName "kube-api-access-84d6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:43.671817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.671404    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:43.699256 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.689276    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:43.700778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.700751    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:43.700943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.700928    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84d6l\" (UniqueName: \"kubernetes.io/projected/763c53db-cc32-40fe-a758-67b9d7ccbef9-kube-api-access-84d6l\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:43.716982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.716456    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-xks-ca" pod "763c53db-cc32-40fe-a758-67b9d7ccbef9" (UID: "763c53db-cc32-40fe-a758-67b9d7ccbef9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:43.719360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.718744    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:29:43.719360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.718834    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:29:43.775315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.771676    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6601-b8edec"]
Nov 02 01:29:43.775315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.772759    4851 cpu_manager.go:414] "containerMap: removing container" podUID="763c53db-cc32-40fe-a758-67b9d7ccbef9" containerName="revoke-keys"
Nov 02 01:29:43.775315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.772781    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763c53db-cc32-40fe-a758-67b9d7ccbef9" containerName="revoke-keys"
Nov 02 01:29:43.775315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.772987    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="763c53db-cc32-40fe-a758-67b9d7ccbef9" containerName="revoke-keys"
Nov 02 01:29:43.853383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.820653    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763c53db-cc32-40fe-a758-67b9d7ccbef9-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:43.853594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.838350    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:29:43.888453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.888408    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 01:29:43.889104 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.847847    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:43.893311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.847787    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:43.899630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.899604    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:29:43.903313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.900590    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:29:43.921022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.920992    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vhtnz\""
Nov 02 01:29:43.968514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.968466    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:44.004922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.973983    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:29:44.004922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.976598    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 01:29:44.004922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.976644    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 01:29:44.004922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.996996    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:44.004922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:43.997434    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:29:44.041550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.006060    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:44.041550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.006425    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:29:44.041550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.006724    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:29:44.041550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.006785    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:29:44.041550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.018631    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:29:44.041550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.019068    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:29:44.101326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.085426    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6pd\" (UniqueName: \"kubernetes.io/projected/e5e629ab-dc91-41b3-aa5e-b2a1724f7a00-kube-api-access-2n6pd\") pod \"test-curl-6601-b8edec\" (UID: \"e5e629ab-dc91-41b3-aa5e-b2a1724f7a00\") " pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:29:44.154374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.153434    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:29:44.154374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.153493    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:29:44.191433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.182180    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6601-b8edec"]
Nov 02 01:29:44.191433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.182318    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:44.191433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.183728    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:29:44.191433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.184523    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:29:44.262537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.208350    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6pd\" (UniqueName: \"kubernetes.io/projected/e5e629ab-dc91-41b3-aa5e-b2a1724f7a00-kube-api-access-2n6pd\") pod \"test-curl-6601-b8edec\" (UID: \"e5e629ab-dc91-41b3-aa5e-b2a1724f7a00\") " pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:29:44.295013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.292800    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:29:44.343611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.332240    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:29:44.343611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.339524    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6pd\" (UniqueName: \"kubernetes.io/projected/e5e629ab-dc91-41b3-aa5e-b2a1724f7a00-kube-api-access-2n6pd\") pod \"test-curl-6601-b8edec\" (UID: \"e5e629ab-dc91-41b3-aa5e-b2a1724f7a00\") " pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:29:44.440099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.440048    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:29:44.479047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.479016    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:29:44.479333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.479282    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 01:29:44.621684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.597072    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:44.621684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.598243    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:29:44.656431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.656384    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:29:44.698463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.698363    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:44.765617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.698847    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:29:44.765890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.765860    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn" event={"ID":"763c53db-cc32-40fe-a758-67b9d7ccbef9","Type":"ContainerDied","Data":"aafaba06b59e83e5f7aee64241a058d57956ec32e14d9be620a6a944f4bc57e7"}
Nov 02 01:29:44.766023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.766009    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aafaba06b59e83e5f7aee64241a058d57956ec32e14d9be620a6a944f4bc57e7"
Nov 02 01:29:44.777541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.699694    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:29:44.803151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.803122    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"
Nov 02 01:29:44.832003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.831978    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:29:44.842460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.842430    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:29:44.870450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.870402    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 01:29:45.035167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:44.902697    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.180331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.003344    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.180331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.004599    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.180331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.006391    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.180331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.027457    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.180331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.102537    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 01:29:45.180331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.159680    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 01:29:45.184105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.184070    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.311666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.311570    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.334438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.334394    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.335673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.335581    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:29:45.335673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.335623    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:29:45.434579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.433951    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:29:45.434579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.434004    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 01:29:45.602391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.601191    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:45.630312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.607940    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.676316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.669510    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.771089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.771047    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.797317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.778095    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.867365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.857743    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:29:45.867365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.857804    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 01:29:45.867365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.858162    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:45.963317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.917361    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 01:29:45.963317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.917418    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 01:29:45.988140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:45.988107    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:29:46.074318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.060069    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:46.230313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.205678    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:29:46.231502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.231456    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:29:46.309783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.309737    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.343440    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.363492    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.363529    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.363552    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.363573    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.363593    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.575471    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.413727    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.685381    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.619404    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.167838    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.414076    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.245047    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.775947    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:46.775995    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.168346    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.273379    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.414592    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.293841    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.414636    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 01:29:47.446436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.230023    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Nov 02 01:29:47.524317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.480800    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:29:47.842317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.535177    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:29:47.842317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.824632    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 01:29:47.842317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.812408    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:47.863347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.848161    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:29:47.863347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.848214    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 01:29:47.918340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.913064    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:47.958441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.958403    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 01:29:47.958739 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:47.958698    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 01:29:48.210549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.210481    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:29:48.210549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.210540    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:29:48.251909 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:29:48.251868    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:29:48.253924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.252185    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:29:48.518362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.496708    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:29:48.520132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.511558    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:29:48.521035 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.511594    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:29:48.521393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.511616    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:29:48.528312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.511636    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:29:48.528967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.511655    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:29:48.543660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.511674    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:29:48.544452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.511692    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:29:48.544871 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.511713    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:29:48.545195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.511998    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:29:48.545383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.545355    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 01:29:48.561839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.518351    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:29:48.623338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.617555    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"]
Nov 02 01:29:48.656317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.649801    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:48.668315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.662829    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:29:48.668315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.662959    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:29:48.668315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.665541    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:29:48.668315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.667218    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:29:48.716912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.716855    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:29:48.721857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.721827    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"]
Nov 02 01:29:48.801959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.779354    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3978ac42-qg62n\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:48.902403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.839913    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:29:48.902762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.902718    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 01:29:48.965843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.965792    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 01:29:48.966125 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:48.966080    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 01:29:49.031262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.031221    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3978ac42-qg62n\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:49.034467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.034440    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t76p9\" (UniqueName: \"kubernetes.io/projected/85f074b2-0dca-42a3-9eba-2b1480122136-kube-api-access-t76p9\") pod \"maas-api-revoke-sub-3978ac42-qg62n\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:49.276319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.247899    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:29:49.277089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.277059    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:29:49.277350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.277335    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:29:49.277539 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.277526    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:29:49.277719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.277705    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:29:49.277879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.277866    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:29:49.278263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.278245    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:29:49.305483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.305449    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:29:49.337233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.337195    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6601-b8edec" event={"ID":"e5e629ab-dc91-41b3-aa5e-b2a1724f7a00","Type":"ContainerStarted","Data":"40fcedb3ada59902f75f9de5fd5ba58af689789c3d2a389b117615fd3f8ef671"}
Nov 02 01:29:49.357947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.357917    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:29:49.358569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.358510    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:29:49.397021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.396982    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3978ac42-qg62n\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:49.397199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.397172    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t76p9\" (UniqueName: \"kubernetes.io/projected/85f074b2-0dca-42a3-9eba-2b1480122136-kube-api-access-t76p9\") pod \"maas-api-revoke-sub-3978ac42-qg62n\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:49.397387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.397368    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3978ac42-qg62n\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:49.399562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.399533    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3978ac42-qg62n\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:49.435620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.434488    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:29:49.435620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.434549    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 01:29:49.437392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.436123    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3978ac42-qg62n\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:49.466336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.463975    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"]
Nov 02 01:29:49.470565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.470502    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t76p9\" (UniqueName: \"kubernetes.io/projected/85f074b2-0dca-42a3-9eba-2b1480122136-kube-api-access-t76p9\") pod \"maas-api-revoke-sub-3978ac42-qg62n\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:49.475439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.475417    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"]
Nov 02 01:29:49.475720 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.475699    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.579050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.579016    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6601-b8edec"]
Nov 02 01:29:49.612971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.612934    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:49.616904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.616857    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78b341c-7p7b8\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.616974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.616950    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78b341c-7p7b8\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.617122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.617100    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2fc\" (UniqueName: \"kubernetes.io/projected/d280a75e-6351-4e05-b522-cbb429cffa85-kube-api-access-cz2fc\") pod \"maas-api-revoke-sub-b78b341c-7p7b8\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.705310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.705172    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:49.735461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.735107    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78b341c-7p7b8\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.735461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.735418    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2fc\" (UniqueName: \"kubernetes.io/projected/d280a75e-6351-4e05-b522-cbb429cffa85-kube-api-access-cz2fc\") pod \"maas-api-revoke-sub-b78b341c-7p7b8\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.735731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.735661    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78b341c-7p7b8\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.737069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.736743    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78b341c-7p7b8\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.765596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.765284    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78b341c-7p7b8\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.797007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.795147    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2fc\" (UniqueName: \"kubernetes.io/projected/d280a75e-6351-4e05-b522-cbb429cffa85-kube-api-access-cz2fc\") pod \"maas-api-revoke-sub-b78b341c-7p7b8\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.834936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.834881    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 01:29:49.835334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.835132    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 01:29:49.840993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.840881    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:49.916337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.900330    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:49.984256 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.955590    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 01:29:49.984256 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:49.955641    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 01:29:50.155439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:50.154454    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 02 01:29:50.155439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:50.154512    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 02 01:29:50.411730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:50.411684    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"]
Nov 02 01:29:50.609852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:50.609825    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:50.611992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:50.611963    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:50.616244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:50.616227    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:29:50.693635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:50.693538    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:50.762837 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:50.762795    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"]
Nov 02 01:29:51.047313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.044639    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:51.047313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.046760    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:51.060656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.060624    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:29:51.202926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.202818    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:51.204424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.204387    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:51.205593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.205573    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:29:51.267407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.263549    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:51.267407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.265063    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:51.267407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.266840    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:29:51.543897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.543813    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:51.658763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.658718    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n" event={"ID":"85f074b2-0dca-42a3-9eba-2b1480122136","Type":"ContainerStarted","Data":"d09c8252604f9385343e29e9420a8d2d4f636527e331d9752ab6a5c3f7759db0"}
Nov 02 01:29:51.659358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.659331    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n" event={"ID":"85f074b2-0dca-42a3-9eba-2b1480122136","Type":"ContainerStarted","Data":"4e5abff4ae508cbb048300d1c490c54d474fded3579f2956400964c964316766"}
Nov 02 01:29:51.665850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.665764    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8" event={"ID":"d280a75e-6351-4e05-b522-cbb429cffa85","Type":"ContainerStarted","Data":"f9c8eeab3613744a756df86a673ee0a15fd6667bf6860a70e122c8c05b2269b0"}
Nov 02 01:29:51.682933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:51.682879    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n" podStartSLOduration=7.682862965 podStartE2EDuration="7.682862965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:51.680368849 +0000 UTC m=+896.985301712" watchObservedRunningTime="2026-11-02 01:29:51.682862965 +0000 UTC m=+896.987795835"
Nov 02 01:29:52.342389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:52.342311    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:52.677485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:52.677452    4851 generic.go:356] "Generic (PLEG): container finished" podID="85f074b2-0dca-42a3-9eba-2b1480122136" containerID="d09c8252604f9385343e29e9420a8d2d4f636527e331d9752ab6a5c3f7759db0" exitCode=0
Nov 02 01:29:52.677954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:52.677535    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n" event={"ID":"85f074b2-0dca-42a3-9eba-2b1480122136","Type":"ContainerDied","Data":"d09c8252604f9385343e29e9420a8d2d4f636527e331d9752ab6a5c3f7759db0"}
Nov 02 01:29:52.679970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:52.679944    4851 generic.go:356] "Generic (PLEG): container finished" podID="d280a75e-6351-4e05-b522-cbb429cffa85" containerID="86895bb91f8d1c6e40e33afa1f35e3329624fa095e9bfcaa00fdf6102f789b9b" exitCode=0
Nov 02 01:29:52.680109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:52.680047    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8" event={"ID":"d280a75e-6351-4e05-b522-cbb429cffa85","Type":"ContainerDied","Data":"86895bb91f8d1c6e40e33afa1f35e3329624fa095e9bfcaa00fdf6102f789b9b"}
Nov 02 01:29:53.232222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:53.232188    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:53.921894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:53.921857    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:53.976790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:53.976760    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:54.020203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.020157    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-openshift-ca\") pod \"d280a75e-6351-4e05-b522-cbb429cffa85\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") "
Nov 02 01:29:54.020401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.020244    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-xks-ca\") pod \"d280a75e-6351-4e05-b522-cbb429cffa85\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") "
Nov 02 01:29:54.020467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.020398    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d280a75e-6351-4e05-b522-cbb429cffa85-kube-api-access-cz2fc\" (UniqueName: \"kubernetes.io/projected/d280a75e-6351-4e05-b522-cbb429cffa85-kube-api-access-cz2fc\") pod \"d280a75e-6351-4e05-b522-cbb429cffa85\" (UID: \"d280a75e-6351-4e05-b522-cbb429cffa85\") "
Nov 02 01:29:54.020578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.020549    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-openshift-ca" pod "d280a75e-6351-4e05-b522-cbb429cffa85" (UID: "d280a75e-6351-4e05-b522-cbb429cffa85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:54.021139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.021118    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:54.023502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.023449    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-xks-ca" pod "d280a75e-6351-4e05-b522-cbb429cffa85" (UID: "d280a75e-6351-4e05-b522-cbb429cffa85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:54.026587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.026557    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d280a75e-6351-4e05-b522-cbb429cffa85-kube-api-access-cz2fc" pod "d280a75e-6351-4e05-b522-cbb429cffa85" (UID: "d280a75e-6351-4e05-b522-cbb429cffa85"). InnerVolumeSpecName "kube-api-access-cz2fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:54.074059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.073950    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:54.123051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.122005    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-openshift-ca\") pod \"85f074b2-0dca-42a3-9eba-2b1480122136\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") "
Nov 02 01:29:54.123051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.122172    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-xks-ca\") pod \"85f074b2-0dca-42a3-9eba-2b1480122136\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") "
Nov 02 01:29:54.123051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.122446    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f074b2-0dca-42a3-9eba-2b1480122136-kube-api-access-t76p9\" (UniqueName: \"kubernetes.io/projected/85f074b2-0dca-42a3-9eba-2b1480122136-kube-api-access-t76p9\") pod \"85f074b2-0dca-42a3-9eba-2b1480122136\" (UID: \"85f074b2-0dca-42a3-9eba-2b1480122136\") "
Nov 02 01:29:54.156190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.156076    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d280a75e-6351-4e05-b522-cbb429cffa85-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:54.156190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.156144    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2fc\" (UniqueName: \"kubernetes.io/projected/d280a75e-6351-4e05-b522-cbb429cffa85-kube-api-access-cz2fc\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:54.186502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.186433    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-openshift-ca" pod "85f074b2-0dca-42a3-9eba-2b1480122136" (UID: "85f074b2-0dca-42a3-9eba-2b1480122136"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:29:54.186502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.186474    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f074b2-0dca-42a3-9eba-2b1480122136-kube-api-access-t76p9" pod "85f074b2-0dca-42a3-9eba-2b1480122136" (UID: "85f074b2-0dca-42a3-9eba-2b1480122136"). InnerVolumeSpecName "kube-api-access-t76p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:29:54.189410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.189369    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-xks-ca" pod "85f074b2-0dca-42a3-9eba-2b1480122136" (UID: "85f074b2-0dca-42a3-9eba-2b1480122136"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:29:54.257885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.257803    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"]
Nov 02 01:29:54.258148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.257926    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:54.258262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.258162    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f074b2-0dca-42a3-9eba-2b1480122136-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:54.258262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.258182    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t76p9\" (UniqueName: \"kubernetes.io/projected/85f074b2-0dca-42a3-9eba-2b1480122136-kube-api-access-t76p9\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:29:54.260394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.258910    4851 cpu_manager.go:414] "containerMap: removing container" podUID="85f074b2-0dca-42a3-9eba-2b1480122136" containerName="revoke-keys"
Nov 02 01:29:54.260394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.258951    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f074b2-0dca-42a3-9eba-2b1480122136" containerName="revoke-keys"
Nov 02 01:29:54.260394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.259020    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d280a75e-6351-4e05-b522-cbb429cffa85" containerName="revoke-keys"
Nov 02 01:29:54.260394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.259030    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d280a75e-6351-4e05-b522-cbb429cffa85" containerName="revoke-keys"
Nov 02 01:29:54.260394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.259214    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d280a75e-6351-4e05-b522-cbb429cffa85" containerName="revoke-keys"
Nov 02 01:29:54.260394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.259246    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f074b2-0dca-42a3-9eba-2b1480122136" containerName="revoke-keys"
Nov 02 01:29:54.266773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.265979    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.272660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.272635    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"]
Nov 02 01:29:54.292926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.292902    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"]
Nov 02 01:29:54.310174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.310150    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.316918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.315681    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"]
Nov 02 01:29:54.359477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.359424    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.359634 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.359520    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.359634 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.359550    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.359766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.359729    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.359825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.359795    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.359880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.359847    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.360050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.360024    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7p4\" (UniqueName: \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-kube-api-access-vh7p4\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.360192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.360173    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.360268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.360211    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.428978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.428942    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"]
Nov 02 01:29:54.441051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.441027    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.457429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.457401    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"]
Nov 02 01:29:54.461577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.461508    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.461706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.461590    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.461706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.461644    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.461706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.461675    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.461706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.461700    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4tjl\" (UniqueName: \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-kube-api-access-n4tjl\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.461932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.461743    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.461932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.461774    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.461932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.461850    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.461932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.461880    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.461932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.461924    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7p4\" (UniqueName: \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-kube-api-access-vh7p4\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.462194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.462001    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.462194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.462034    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.462194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.462086    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.462194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.462116    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.462194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.462179    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.463175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.462744    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.463175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.462808    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.463175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.462844    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.463175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.463055    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.463495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.463239    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.463495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.463322    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.463495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.463364    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.463765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.463746    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.466122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.466085    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.466122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.466117    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.477954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.477930    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.478132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.478111    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7p4\" (UniqueName: \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-kube-api-access-vh7p4\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.564781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.564679    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.564781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.564730    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4tjl\" (UniqueName: \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-kube-api-access-n4tjl\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.564781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.564768    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.565065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.564791    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.565065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.564854    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.565065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.564880    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.565065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.564908    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-605ce8fc-6qbtg\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.565065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.564984    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-605ce8fc-6qbtg\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.565065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.565017    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.565065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.565051    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.565431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.565089    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.565431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.565222    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwx4\" (UniqueName: \"kubernetes.io/projected/7fc61da2-91a6-4eba-9b66-f8f375b61055-kube-api-access-smwx4\") pod \"maas-api-revoke-sub-605ce8fc-6qbtg\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.565431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.565268    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.565717 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.565646    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.565807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.565718    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.565807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.565752    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.566127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.566099    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.567717 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.567696    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.568412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.568396    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.577945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.577923    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.578082 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.578055    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4tjl\" (UniqueName: \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-kube-api-access-n4tjl\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.593036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.593011    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:54.631770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.631738    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:54.667012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.666975    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-605ce8fc-6qbtg\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.667119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.667080    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-605ce8fc-6qbtg\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.667373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.667227    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwx4\" (UniqueName: \"kubernetes.io/projected/7fc61da2-91a6-4eba-9b66-f8f375b61055-kube-api-access-smwx4\") pod \"maas-api-revoke-sub-605ce8fc-6qbtg\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.668396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.668356    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-605ce8fc-6qbtg\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.683544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.680380    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwx4\" (UniqueName: \"kubernetes.io/projected/7fc61da2-91a6-4eba-9b66-f8f375b61055-kube-api-access-smwx4\") pod \"maas-api-revoke-sub-605ce8fc-6qbtg\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.684379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.684240    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-605ce8fc-6qbtg\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.711164 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.710898    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"
Nov 02 01:29:54.713315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.712502    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n" event={"ID":"85f074b2-0dca-42a3-9eba-2b1480122136","Type":"ContainerDied","Data":"4e5abff4ae508cbb048300d1c490c54d474fded3579f2956400964c964316766"}
Nov 02 01:29:54.713315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.712571    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e5abff4ae508cbb048300d1c490c54d474fded3579f2956400964c964316766"
Nov 02 01:29:54.717683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.717412    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8" event={"ID":"d280a75e-6351-4e05-b522-cbb429cffa85","Type":"ContainerDied","Data":"f9c8eeab3613744a756df86a673ee0a15fd6667bf6860a70e122c8c05b2269b0"}
Nov 02 01:29:54.717683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.717466    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9c8eeab3613744a756df86a673ee0a15fd6667bf6860a70e122c8c05b2269b0"
Nov 02 01:29:54.717683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.717539    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"
Nov 02 01:29:54.766657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.766630    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:29:54.897400 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.897372    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"]
Nov 02 01:29:54.902180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.902150    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:54.902262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.902212    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:54.902262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.902258    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:54.975605 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.975556    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"]
Nov 02 01:29:54.978733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.978696    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:54.978820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.978764    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:54.978820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:54.978794    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:29:55.020209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.020185    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"]
Nov 02 01:29:55.026729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.026702    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.036783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.036766    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"]
Nov 02 01:29:55.075086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.075061    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:29:55.076995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.076975    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:29:55.137361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.133619    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:55.235089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.232104    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bcb9951-gq9dc\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.235089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.232337    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bcb9951-gq9dc\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.235089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.232457    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrxf5\" (UniqueName: \"kubernetes.io/projected/0e1766f9-294d-4345-b7c8-7018f1b36581-kube-api-access-mrxf5\") pod \"maas-api-revoke-sub-7bcb9951-gq9dc\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.237524 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:55.237464    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc61da2_91a6_4eba_9b66_f8f375b61055.slice/crio-fb37a0948799c29652a959ed82e485e7735120068d309e66433b7fe7cdd09020 WatchSource:0}: Error finding container fb37a0948799c29652a959ed82e485e7735120068d309e66433b7fe7cdd09020: Status 404 returned error can't find the container with id fb37a0948799c29652a959ed82e485e7735120068d309e66433b7fe7cdd09020
Nov 02 01:29:55.285792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.285760    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"]
Nov 02 01:29:55.334454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.334422    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bcb9951-gq9dc\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.334601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.334542    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bcb9951-gq9dc\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.334661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.334602    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrxf5\" (UniqueName: \"kubernetes.io/projected/0e1766f9-294d-4345-b7c8-7018f1b36581-kube-api-access-mrxf5\") pod \"maas-api-revoke-sub-7bcb9951-gq9dc\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.335587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.335559    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bcb9951-gq9dc\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.337349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.337323    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bcb9951-gq9dc\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.349656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.349622    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrxf5\" (UniqueName: \"kubernetes.io/projected/0e1766f9-294d-4345-b7c8-7018f1b36581-kube-api-access-mrxf5\") pod \"maas-api-revoke-sub-7bcb9951-gq9dc\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.352594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.352572    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:29:55.652059 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:29:55.652028    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1766f9_294d_4345_b7c8_7018f1b36581.slice/crio-cf6bde6c54256b84943f5c44c3388d25edc18884e1264f328748edad5db40449 WatchSource:0}: Error finding container cf6bde6c54256b84943f5c44c3388d25edc18884e1264f328748edad5db40449: Status 404 returned error can't find the container with id cf6bde6c54256b84943f5c44c3388d25edc18884e1264f328748edad5db40449
Nov 02 01:29:55.657071 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.657039    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"]
Nov 02 01:29:55.740382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.740341    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg" event={"ID":"7fc61da2-91a6-4eba-9b66-f8f375b61055","Type":"ContainerStarted","Data":"fb37a0948799c29652a959ed82e485e7735120068d309e66433b7fe7cdd09020"}
Nov 02 01:29:55.742813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.742779    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" event={"ID":"e8c4b040-6e11-4236-b7f2-9438685e5ba3","Type":"ContainerStarted","Data":"530fa39b156027983bdd36ab12fa28256897addba275241680b867749d73f78c"}
Nov 02 01:29:55.743048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.743031    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" event={"ID":"e8c4b040-6e11-4236-b7f2-9438685e5ba3","Type":"ContainerStarted","Data":"43ae94b8731a8cb47a9f65381778af2fcc4308d4ad92c951a2eac0328f3d6465"}
Nov 02 01:29:55.763320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.754085    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" event={"ID":"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e","Type":"ContainerStarted","Data":"3680c7f5c4e5145d060355918aac14b95c1c0ee3bc619e08f96049285c927404"}
Nov 02 01:29:55.763320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.754149    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" event={"ID":"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e","Type":"ContainerStarted","Data":"87b57d8f646216aa17a616530878f6b5cdcfcde21b648730b3e1b484214f1a53"}
Nov 02 01:29:55.773319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.766124    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6601-b8edec" event={"ID":"e5e629ab-dc91-41b3-aa5e-b2a1724f7a00","Type":"ContainerStarted","Data":"2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214"}
Nov 02 01:29:55.797404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.797349    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podStartSLOduration=1.797335058 podStartE2EDuration="1.797335058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:55.782251817 +0000 UTC m=+901.087184643" watchObservedRunningTime="2026-11-02 01:29:55.797335058 +0000 UTC m=+901.102267883"
Nov 02 01:29:55.797626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.797591    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc" event={"ID":"0e1766f9-294d-4345-b7c8-7018f1b36581","Type":"ContainerStarted","Data":"cf6bde6c54256b84943f5c44c3388d25edc18884e1264f328748edad5db40449"}
Nov 02 01:29:55.822729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.822675    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podStartSLOduration=1.822662019 podStartE2EDuration="1.822662019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:55.818441618 +0000 UTC m=+901.123374444" watchObservedRunningTime="2026-11-02 01:29:55.822662019 +0000 UTC m=+901.127594844"
Nov 02 01:29:55.867323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:55.861378    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6601-b8edec" podStartSLOduration=5.90627595 podStartE2EDuration="12.861356854s" totalImagesPullingTime="6.955080904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:55.853905371 +0000 UTC m=+901.158838197" watchObservedRunningTime="2026-11-02 01:29:55.861356854 +0000 UTC m=+901.166289681"
Nov 02 01:29:56.194853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:56.194822    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:56.627084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:56.607595    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:29:56.627084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:56.618348    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:29:56.627084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:56.618427    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:29:56.635171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:56.633778    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:29:56.635171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:56.634879    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:29:56.635171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:56.634928    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:29:57.167379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.158753    4851 generic.go:356] "Generic (PLEG): container finished" podID="7fc61da2-91a6-4eba-9b66-f8f375b61055" containerID="959be49370e30c729c9c9b151c6ce9d2929db3e927f19f8b348b65bacb0373b8" exitCode=0
Nov 02 01:29:57.340510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.340365    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg" event={"ID":"7fc61da2-91a6-4eba-9b66-f8f375b61055","Type":"ContainerDied","Data":"959be49370e30c729c9c9b151c6ce9d2929db3e927f19f8b348b65bacb0373b8"}
Nov 02 01:29:57.360510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.360470    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc" event={"ID":"0e1766f9-294d-4345-b7c8-7018f1b36581","Type":"ContainerStarted","Data":"65ebcc233b36327449236c5eebf39a66c7506d1f6c0786f5f800f1290b0d8af4"}
Nov 02 01:29:57.416104 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.416026    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:57.418183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.418086    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:29:57.454274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.434379    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:57.619356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.616392    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:57.620092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.619682    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:57.674371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.658327    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:29:57.674371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.658396    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:29:57.674371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.658808    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:29:57.674371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.658845    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:29:57.795240 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.795189    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:57.824319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.815372    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:29:57.906847 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:57.906799    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:29:58.023376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.023261    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:58.244482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.244432    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:29:58.245922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.245889    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:58.246422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.246399    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:29:58.435729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.416898    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:58.435729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.423518    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:29:58.482319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.468634    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:58.482319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.468999    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:29:58.544666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.503265    4851 generic.go:356] "Generic (PLEG): container finished" podID="0e1766f9-294d-4345-b7c8-7018f1b36581" containerID="65ebcc233b36327449236c5eebf39a66c7506d1f6c0786f5f800f1290b0d8af4" exitCode=0
Nov 02 01:29:58.557346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.545248    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:29:58.557346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.545764    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:29:58.557346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.545805    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc" event={"ID":"0e1766f9-294d-4345-b7c8-7018f1b36581","Type":"ContainerDied","Data":"65ebcc233b36327449236c5eebf39a66c7506d1f6c0786f5f800f1290b0d8af4"}
Nov 02 01:29:58.691373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.617263    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:58.691373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.617709    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:29:58.691373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.657234    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:58.691373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.657592    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:29:58.730253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.729773    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:29:58.730253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.729828    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:29:58.827236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.824393    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:58.827236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.824770    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:29:58.827236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.825080    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:29:58.827236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.825134    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:29:58.937497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:58.937467    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:29:59.036376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.024873    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:59.109316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.091170    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:29:59.322376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.304504    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:29:59.662357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.647241    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:29:59.739794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.307373    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:29:59.739794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.737194    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:59.758316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.741351    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:59.792405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.792359    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:29:59.813324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.812552    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:29:59.817315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.813872    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded"
Nov 02 01:29:59.913053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.307445    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:30:00.000353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.843020    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:30:00.045318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.027429    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:00.045318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.861751    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:30:00.045318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.029103    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:30:00.045318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.894540    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:30:00.045318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.029167    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:30:00.330061 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:00.329963    4851 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.476s"
Nov 02 01:30:00.330514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:29:59.843260    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:30:00.448320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.330767    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:30:00.448778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.448756    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:30:00.449035 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.449020    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:30:00.449248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.449235    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:30:00.646881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.646779    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:00.857325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.856069    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:00.884151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.884091    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:30:00.991321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.898010    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:00.991321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.909723    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:30:00.991321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.909873    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:00.991321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.983413    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:30:00.991321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.983530    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:30:00.991321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:00.983588    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:30:01.164478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:01.164429    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:30:01.271124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:01.271012    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:30:01.435453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:01.430411    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:01.541978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:01.430491    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:01.759560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:01.756909    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:30:01.820869 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:01.820772    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:30:01.821343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:01.821104    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:30:01.962356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:01.931401    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:30:01.962356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:01.931466    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:30:02.084873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.084782    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:30:02.085128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.085096    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:30:02.119631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.119595    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:30:02.119954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.119916    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:30:02.369370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.325840    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:30:02.425317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.424545    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:30:02.497340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.474890    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:02.510321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.509408    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:02.510321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.509876    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:30:02.510321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.510230    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:30:02.515313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.511002    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:30:02.515313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.511276    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:30:02.515313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.511558    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:02.515313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.511824    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:30:02.683559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.673389    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:02.701863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.697039    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:30:02.701863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.697126    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:30:02.827361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.827228    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:30:02.827361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.827281    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:30:02.827636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.827373    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:02.827636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.827396    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:02.878318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.857695    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 02 01:30:02.895712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.882632    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:02.897448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.897399    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:30:02.922318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.897939    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:30:02.940816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.940712    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded"
Nov 02 01:30:02.941132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:02.941106    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Nov 02 01:30:03.135277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.135232    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-openshift-ca\") pod \"0e1766f9-294d-4345-b7c8-7018f1b36581\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") "
Nov 02 01:30:03.175317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.143089    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-openshift-ca\") pod \"7fc61da2-91a6-4eba-9b66-f8f375b61055\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") "
Nov 02 01:30:03.175661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.175630    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1766f9-294d-4345-b7c8-7018f1b36581-kube-api-access-mrxf5\" (UniqueName: \"kubernetes.io/projected/0e1766f9-294d-4345-b7c8-7018f1b36581-kube-api-access-mrxf5\") pod \"0e1766f9-294d-4345-b7c8-7018f1b36581\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") "
Nov 02 01:30:03.175972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.175953    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc61da2-91a6-4eba-9b66-f8f375b61055-kube-api-access-smwx4\" (UniqueName: \"kubernetes.io/projected/7fc61da2-91a6-4eba-9b66-f8f375b61055-kube-api-access-smwx4\") pod \"7fc61da2-91a6-4eba-9b66-f8f375b61055\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") "
Nov 02 01:30:03.176117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.176098    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-xks-ca\") pod \"0e1766f9-294d-4345-b7c8-7018f1b36581\" (UID: \"0e1766f9-294d-4345-b7c8-7018f1b36581\") "
Nov 02 01:30:03.176355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.176336    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-xks-ca\") pod \"7fc61da2-91a6-4eba-9b66-f8f375b61055\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") "
Nov 02 01:30:03.256468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.256429    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:03.265323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.256720    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:03.373696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.373657    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-xks-ca" pod "7fc61da2-91a6-4eba-9b66-f8f375b61055" (UID: "7fc61da2-91a6-4eba-9b66-f8f375b61055"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:30:03.482701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.482599    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:03.502378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.494334    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-openshift-ca" pod "0e1766f9-294d-4345-b7c8-7018f1b36581" (UID: "0e1766f9-294d-4345-b7c8-7018f1b36581"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:03.502378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.496543    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded"
Nov 02 01:30:03.519324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.511527    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-openshift-ca" pod "7fc61da2-91a6-4eba-9b66-f8f375b61055" (UID: "7fc61da2-91a6-4eba-9b66-f8f375b61055"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:03.519324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.512728    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:30:03.529325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.524393    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:30:03.549327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.530459    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-xks-ca" pod "0e1766f9-294d-4345-b7c8-7018f1b36581" (UID: "0e1766f9-294d-4345-b7c8-7018f1b36581"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:30:03.549327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.531607    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:03.549327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.535393    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:03.571368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.557399    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:30:03.571368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.557784    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:30:03.571368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.558052    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:03.571368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.558276    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:03.605600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.600114    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:03.605600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.600154    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc61da2-91a6-4eba-9b66-f8f375b61055-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:03.605600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.600170    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e1766f9-294d-4345-b7c8-7018f1b36581-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:03.605600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.602489    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:30:03.605600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.602538    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:30:03.720363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.680281    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:03.720363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.681272    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:30:03.720363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.681335    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:30:03.740428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.740322    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:03.815317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.813233    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"
Nov 02 01:30:03.842682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.842647    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1766f9-294d-4345-b7c8-7018f1b36581-kube-api-access-mrxf5" pod "0e1766f9-294d-4345-b7c8-7018f1b36581" (UID: "0e1766f9-294d-4345-b7c8-7018f1b36581"). InnerVolumeSpecName "kube-api-access-mrxf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:03.851526 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:03.851494    4851 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.597s"
Nov 02 01:30:03.923317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.882670    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc" event={"ID":"0e1766f9-294d-4345-b7c8-7018f1b36581","Type":"ContainerDied","Data":"cf6bde6c54256b84943f5c44c3388d25edc18884e1264f328748edad5db40449"}
Nov 02 01:30:03.923317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.882745    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf6bde6c54256b84943f5c44c3388d25edc18884e1264f328748edad5db40449"
Nov 02 01:30:03.923317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.882763    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg" event={"ID":"7fc61da2-91a6-4eba-9b66-f8f375b61055","Type":"ContainerDied","Data":"fb37a0948799c29652a959ed82e485e7735120068d309e66433b7fe7cdd09020"}
Nov 02 01:30:03.923317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.882791    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb37a0948799c29652a959ed82e485e7735120068d309e66433b7fe7cdd09020"
Nov 02 01:30:03.923317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.889891    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:03.923317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.919425    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:30:03.923317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.920593    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc61da2-91a6-4eba-9b66-f8f375b61055-kube-api-access-smwx4" pod "7fc61da2-91a6-4eba-9b66-f8f375b61055" (UID: "7fc61da2-91a6-4eba-9b66-f8f375b61055"). InnerVolumeSpecName "kube-api-access-smwx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:03.924192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.874036    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"
Nov 02 01:30:03.975796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.962616    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc61da2-91a6-4eba-9b66-f8f375b61055-kube-api-access-smwx4\" (UniqueName: \"kubernetes.io/projected/7fc61da2-91a6-4eba-9b66-f8f375b61055-kube-api-access-smwx4\") pod \"7fc61da2-91a6-4eba-9b66-f8f375b61055\" (UID: \"7fc61da2-91a6-4eba-9b66-f8f375b61055\") "
Nov 02 01:30:03.975796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.963605    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrxf5\" (UniqueName: \"kubernetes.io/projected/0e1766f9-294d-4345-b7c8-7018f1b36581-kube-api-access-mrxf5\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:03.975796 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:03.963707    4851 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/7fc61da2-91a6-4eba-9b66-f8f375b61055/volumes/kubernetes.io~projected/kube-api-access-smwx4
Nov 02 01:30:03.975796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.963764    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc61da2-91a6-4eba-9b66-f8f375b61055-kube-api-access-smwx4" pod "7fc61da2-91a6-4eba-9b66-f8f375b61055" (UID: "7fc61da2-91a6-4eba-9b66-f8f375b61055"). InnerVolumeSpecName "kube-api-access-smwx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:03.975796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.964238    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:03.975796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:03.965383    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded"
Nov 02 01:30:04.067341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.028915    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:04.095315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.076224    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smwx4\" (UniqueName: \"kubernetes.io/projected/7fc61da2-91a6-4eba-9b66-f8f375b61055-kube-api-access-smwx4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:04.095315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.080906    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"]
Nov 02 01:30:04.149316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.145458    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0e1766f9-294d-4345-b7c8-7018f1b36581" containerName="revoke-keys"
Nov 02 01:30:04.149316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.145580    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1766f9-294d-4345-b7c8-7018f1b36581" containerName="revoke-keys"
Nov 02 01:30:04.149316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.145614    4851 cpu_manager.go:414] "containerMap: removing container" podUID="7fc61da2-91a6-4eba-9b66-f8f375b61055" containerName="revoke-keys"
Nov 02 01:30:04.149316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.145622    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc61da2-91a6-4eba-9b66-f8f375b61055" containerName="revoke-keys"
Nov 02 01:30:04.149316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.146404    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1766f9-294d-4345-b7c8-7018f1b36581" containerName="revoke-keys"
Nov 02 01:30:04.149316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.146427    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc61da2-91a6-4eba-9b66-f8f375b61055" containerName="revoke-keys"
Nov 02 01:30:04.233436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.233401    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"]
Nov 02 01:30:04.234407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.234383    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"
Nov 02 01:30:04.253389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.246408    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:30:04.253389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.246534    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:30:04.253389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.246657    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:30:04.309373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.284306    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac436667-7100-41f1-b3ca-1bc1029f3266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx\" (UID: \"ac436667-7100-41f1-b3ca-1bc1029f3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"
Nov 02 01:30:04.309777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.309744    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxbx\" (UniqueName: \"kubernetes.io/projected/ac436667-7100-41f1-b3ca-1bc1029f3266-kube-api-access-pjxbx\") pod \"maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx\" (UID: \"ac436667-7100-41f1-b3ca-1bc1029f3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"
Nov 02 01:30:04.314923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.314897    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"]
Nov 02 01:30:04.338410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.315677    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"
Nov 02 01:30:04.413184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.413146    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6jmk\" (UniqueName: \"kubernetes.io/projected/27fb8c50-9ff8-4be1-b961-6f2474b48804-kube-api-access-g6jmk\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg\" (UID: \"27fb8c50-9ff8-4be1-b961-6f2474b48804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"
Nov 02 01:30:04.413590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.413561    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac436667-7100-41f1-b3ca-1bc1029f3266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx\" (UID: \"ac436667-7100-41f1-b3ca-1bc1029f3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"
Nov 02 01:30:04.416319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.413810    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxbx\" (UniqueName: \"kubernetes.io/projected/ac436667-7100-41f1-b3ca-1bc1029f3266-kube-api-access-pjxbx\") pod \"maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx\" (UID: \"ac436667-7100-41f1-b3ca-1bc1029f3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"
Nov 02 01:30:04.416506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.416484    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27fb8c50-9ff8-4be1-b961-6f2474b48804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg\" (UID: \"27fb8c50-9ff8-4be1-b961-6f2474b48804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"
Nov 02 01:30:04.425699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.425670    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac436667-7100-41f1-b3ca-1bc1029f3266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx\" (UID: \"ac436667-7100-41f1-b3ca-1bc1029f3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"
Nov 02 01:30:04.501786 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.501747    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"
Nov 02 01:30:04.527316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.518280    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27fb8c50-9ff8-4be1-b961-6f2474b48804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg\" (UID: \"27fb8c50-9ff8-4be1-b961-6f2474b48804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"
Nov 02 01:30:04.527316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.518563    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6jmk\" (UniqueName: \"kubernetes.io/projected/27fb8c50-9ff8-4be1-b961-6f2474b48804-kube-api-access-g6jmk\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg\" (UID: \"27fb8c50-9ff8-4be1-b961-6f2474b48804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"
Nov 02 01:30:04.527316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.519840    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27fb8c50-9ff8-4be1-b961-6f2474b48804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg\" (UID: \"27fb8c50-9ff8-4be1-b961-6f2474b48804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"
Nov 02 01:30:04.527316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.519926    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:04.534553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.534485    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:04.537050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.535679    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:04.537050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.535951    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:04.583423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.581409    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:30:04.583423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.581737    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:30:04.583423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.581981    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:30:04.583423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.582202    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:04.586825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.586122    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:04.586825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.586177    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:04.621332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.605810    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:30:04.621332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.605867    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:30:04.621332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.611593    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6jmk\" (UniqueName: \"kubernetes.io/projected/27fb8c50-9ff8-4be1-b961-6f2474b48804-kube-api-access-g6jmk\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg\" (UID: \"27fb8c50-9ff8-4be1-b961-6f2474b48804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"
Nov 02 01:30:04.655494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.623037    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b27e232-7a42-4130-b455-4973b6627bd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp\" (UID: \"6b27e232-7a42-4130-b455-4973b6627bd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"
Nov 02 01:30:04.656069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.656019    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxxw\" (UniqueName: \"kubernetes.io/projected/6b27e232-7a42-4130-b455-4973b6627bd5-kube-api-access-mqxxw\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp\" (UID: \"6b27e232-7a42-4130-b455-4973b6627bd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"
Nov 02 01:30:04.656989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.656956    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"
Nov 02 01:30:04.693047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.661435    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:30:04.693047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.687590    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:30:04.693047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.662657    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxbx\" (UniqueName: \"kubernetes.io/projected/ac436667-7100-41f1-b3ca-1bc1029f3266-kube-api-access-pjxbx\") pod \"maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx\" (UID: \"ac436667-7100-41f1-b3ca-1bc1029f3266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"
Nov 02 01:30:04.693047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.682801    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:30:04.693047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.662353    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"]
Nov 02 01:30:04.733349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.732537    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"]
Nov 02 01:30:04.733349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.732594    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"]
Nov 02 01:30:04.733349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.732706    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"
Nov 02 01:30:04.802341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.738508    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"]
Nov 02 01:30:04.802341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.760392    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxxw\" (UniqueName: \"kubernetes.io/projected/6b27e232-7a42-4130-b455-4973b6627bd5-kube-api-access-mqxxw\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp\" (UID: \"6b27e232-7a42-4130-b455-4973b6627bd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"
Nov 02 01:30:04.831154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.831117    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b27e232-7a42-4130-b455-4973b6627bd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp\" (UID: \"6b27e232-7a42-4130-b455-4973b6627bd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"
Nov 02 01:30:04.832566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.832537    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b27e232-7a42-4130-b455-4973b6627bd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp\" (UID: \"6b27e232-7a42-4130-b455-4973b6627bd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"
Nov 02 01:30:04.863436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.845014    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"]
Nov 02 01:30:04.951551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.924418    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:30:04.951551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.926995    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:05.018317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.986028    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2\" (UID: \"d68e7f7a-71a8-47c1-9160-1b35c83e22a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"
Nov 02 01:30:05.106675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.103458    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfsb\" (UniqueName: \"kubernetes.io/projected/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-kube-api-access-rvfsb\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2\" (UID: \"d68e7f7a-71a8-47c1-9160-1b35c83e22a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"
Nov 02 01:30:05.108957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.995014    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxxw\" (UniqueName: \"kubernetes.io/projected/6b27e232-7a42-4130-b455-4973b6627bd5-kube-api-access-mqxxw\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp\" (UID: \"6b27e232-7a42-4130-b455-4973b6627bd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"
Nov 02 01:30:05.109493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.995174    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:30:05.109964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:04.992747    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"
Nov 02 01:30:05.131311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.131269    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:30:05.131549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.131536    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:30:05.131652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.131641    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"]
Nov 02 01:30:05.215887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.212316    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:30:05.215887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.212356    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"]
Nov 02 01:30:05.215887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.212491    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.236373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.223990    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:30:05.256363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.240435    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:05.310057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.310030    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:30:05.327954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.321212    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4035642c-gk76s\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.327954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.321433    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4035642c-gk76s\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.327954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.321506    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2\" (UID: \"d68e7f7a-71a8-47c1-9160-1b35c83e22a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"
Nov 02 01:30:05.327954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.321552    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfsb\" (UniqueName: \"kubernetes.io/projected/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-kube-api-access-rvfsb\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2\" (UID: \"d68e7f7a-71a8-47c1-9160-1b35c83e22a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"
Nov 02 01:30:05.327954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.321627    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxncq\" (UniqueName: \"kubernetes.io/projected/f0aff15e-598c-4f9b-955d-d183f7c7dd38-kube-api-access-wxncq\") pod \"maas-api-revoke-sub-4035642c-gk76s\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.327954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.323422    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2\" (UID: \"d68e7f7a-71a8-47c1-9160-1b35c83e22a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"
Nov 02 01:30:05.353899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.353853    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:30:05.369616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.369586    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"
Nov 02 01:30:05.506961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.506905    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4035642c-gk76s\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.507501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.507480    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4035642c-gk76s\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.507767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.507751    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxncq\" (UniqueName: \"kubernetes.io/projected/f0aff15e-598c-4f9b-955d-d183f7c7dd38-kube-api-access-wxncq\") pod \"maas-api-revoke-sub-4035642c-gk76s\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.510311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.510263    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4035642c-gk76s\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.525956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.525363    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:05.600601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.600562    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:05.601426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.601398    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:30:05.607324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.607284    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfsb\" (UniqueName: \"kubernetes.io/projected/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-kube-api-access-rvfsb\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2\" (UID: \"d68e7f7a-71a8-47c1-9160-1b35c83e22a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"
Nov 02 01:30:05.608947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.608922    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:05.609541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.609519    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:30:05.637214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.637186    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:30:05.637478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.637451    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:30:05.683841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.683810    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:30:05.684095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.684067    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:30:05.731266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.731240    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"
Nov 02 01:30:05.805562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.805518    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4035642c-gk76s\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.822132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.822095    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxncq\" (UniqueName: \"kubernetes.io/projected/f0aff15e-598c-4f9b-955d-d183f7c7dd38-kube-api-access-wxncq\") pod \"maas-api-revoke-sub-4035642c-gk76s\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.823938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.823916    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:30:05.879758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.879724    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:05.884824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.884794    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:30:05.920305 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.920264    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:30:05.946768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:05.939574    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:30:06.013833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.013713    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:30:06.021337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.021317    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:30:06.101336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.100120    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:30:06.303151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.302578    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:06.311475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.311265    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6601-b8edec"]
Nov 02 01:30:06.311588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.311505    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6601-b8edec" podUID="e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" containerName="test-curl-6601-b8edec" containerID="cri-o://2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214" gracePeriod=2
Nov 02 01:30:06.320784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.320730    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6601-b8edec"]
Nov 02 01:30:06.323681 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:06.323533    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-b8edec\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:30:06.329494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.329466    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6601-7d895f"]
Nov 02 01:30:06.387009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.386983    4851 cpu_manager.go:414] "containerMap: removing container" podUID="e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" containerName="test-curl-6601-b8edec"
Nov 02 01:30:06.387168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.387153    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" containerName="test-curl-6601-b8edec"
Nov 02 01:30:06.387544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.387526    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" containerName="test-curl-6601-b8edec"
Nov 02 01:30:06.400896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.400873    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:06.403726 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:06.403692    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-b8edec\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:30:06.406370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.406345    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6601-7d895f"]
Nov 02 01:30:06.431068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.431039    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9hts\" (UniqueName: \"kubernetes.io/projected/df26cce1-0a86-46a7-9892-1cd1cc76dd23-kube-api-access-g9hts\") pod \"test-curl-6601-7d895f\" (UID: \"df26cce1-0a86-46a7-9892-1cd1cc76dd23\") " pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:06.687710 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.534794    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9hts\" (UniqueName: \"kubernetes.io/projected/df26cce1-0a86-46a7-9892-1cd1cc76dd23-kube-api-access-g9hts\") pod \"test-curl-6601-7d895f\" (UID: \"df26cce1-0a86-46a7-9892-1cd1cc76dd23\") " pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:06.692734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.629590    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:30:06.692953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.692920    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:30:06.693118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.662047    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9hts\" (UniqueName: \"kubernetes.io/projected/df26cce1-0a86-46a7-9892-1cd1cc76dd23-kube-api-access-g9hts\") pod \"test-curl-6601-7d895f\" (UID: \"df26cce1-0a86-46a7-9892-1cd1cc76dd23\") " pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:06.693262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.662116    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:30:06.693410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.693389    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:30:06.722958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.722929    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:06.799364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.799336    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"]
Nov 02 01:30:06.810098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.810072    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:06.819335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.815437    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"]
Nov 02 01:30:06.871769 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:06.857489    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-b8edec\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:30:06.903358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.900180    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7f62df-hj5dr\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:06.903358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.900570    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5bfb\" (UniqueName: \"kubernetes.io/projected/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-kube-api-access-c5bfb\") pod \"maas-api-revoke-sub-ef7f62df-hj5dr\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:06.903358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.900693    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7f62df-hj5dr\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:06.917977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:06.917892    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"]
Nov 02 01:30:07.002598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.002504    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5bfb\" (UniqueName: \"kubernetes.io/projected/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-kube-api-access-c5bfb\") pod \"maas-api-revoke-sub-ef7f62df-hj5dr\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:07.002747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.002598    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7f62df-hj5dr\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:07.002806 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.002748    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7f62df-hj5dr\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:07.009685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.005060    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7f62df-hj5dr\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:07.085946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.085861    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7f62df-hj5dr\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:07.086846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.086786    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5bfb\" (UniqueName: \"kubernetes.io/projected/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-kube-api-access-c5bfb\") pod \"maas-api-revoke-sub-ef7f62df-hj5dr\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:07.105268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.105244    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"]
Nov 02 01:30:07.122656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.122636    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"]
Nov 02 01:30:07.154314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.150249    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"]
Nov 02 01:30:07.212739 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:07.212705    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0aff15e_598c_4f9b_955d_d183f7c7dd38.slice/crio-ba9dd1e7c0b65913b658065d29c8a88b35fe24d57a066d04076ba4c3566d009d WatchSource:0}: Error finding container ba9dd1e7c0b65913b658065d29c8a88b35fe24d57a066d04076ba4c3566d009d: Status 404 returned error can't find the container with id ba9dd1e7c0b65913b658065d29c8a88b35fe24d57a066d04076ba4c3566d009d
Nov 02 01:30:07.218918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.218890    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:07.227610 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.225362    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" event={"ID":"ac436667-7100-41f1-b3ca-1bc1029f3266","Type":"ContainerStarted","Data":"6c101e61a7377aff41037a05781ab83293c95d24b36014e6890a1b77eca687da"}
Nov 02 01:30:07.295959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.295456    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" event={"ID":"27fb8c50-9ff8-4be1-b961-6f2474b48804","Type":"ContainerStarted","Data":"74e9d31a3a71c7e149f218b948778dba616192f477391e8ac35517717b043e07"}
Nov 02 01:30:07.295959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.295529    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" event={"ID":"27fb8c50-9ff8-4be1-b961-6f2474b48804","Type":"ContainerStarted","Data":"1c70b66a5fb21513c3dafb3929cd89aceb9616fd369970f6b03eb9ae8d23d891"}
Nov 02 01:30:07.306006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.305894    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp" event={"ID":"6b27e232-7a42-4130-b455-4973b6627bd5","Type":"ContainerStarted","Data":"4fc35466c5460c71db9976cd1192197b76069ddfa4733ef7cdac2da8eb4dd8d1"}
Nov 02 01:30:07.310188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.310159    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"]
Nov 02 01:30:07.325522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.325458    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" podStartSLOduration=7.325441944 podStartE2EDuration="7.325441944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:07.317082878 +0000 UTC m=+912.622015699" watchObservedRunningTime="2026-11-02 01:30:07.325441944 +0000 UTC m=+912.630374772"
Nov 02 01:30:07.409891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.408740    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6601-7d895f"]
Nov 02 01:30:07.596468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.596383    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:30:07.596765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.596733    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:30:07.650759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.648612    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:30:07.650759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.648661    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:30:07.697128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.697063    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:07.812511 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:07.811417    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ace0002_2f2b_4bbf_a5e5_8d1fb3e6a905.slice/crio-3fe5cfcdd3f619149bb6ced27616e826ff7c0ab982a0feb4c3e5ca5cdb0cfd44 WatchSource:0}: Error finding container 3fe5cfcdd3f619149bb6ced27616e826ff7c0ab982a0feb4c3e5ca5cdb0cfd44: Status 404 returned error can't find the container with id 3fe5cfcdd3f619149bb6ced27616e826ff7c0ab982a0feb4c3e5ca5cdb0cfd44
Nov 02 01:30:07.843726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.837070    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"]
Nov 02 01:30:07.869920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:07.869868    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:08.289198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.289159    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:08.632700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.605703    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 02 01:30:08.632700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.605756    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 02 01:30:08.657490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.657430    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 02 01:30:08.692347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.661904    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:08.728188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.692772    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 02 01:30:08.728188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.698976    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6601-7d895f" event={"ID":"df26cce1-0a86-46a7-9892-1cd1cc76dd23","Type":"ContainerStarted","Data":"77d742256e00a342fbab27ee8596fcd49df78a49a073a306ac70357871161242"}
Nov 02 01:30:08.728188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.699023    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6601-7d895f" event={"ID":"df26cce1-0a86-46a7-9892-1cd1cc76dd23","Type":"ContainerStarted","Data":"9c1e1b0a064b955289b3405f837fb6833d6439c7c99951612a7f695642e5d254"}
Nov 02 01:30:08.794776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.794009    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" event={"ID":"d68e7f7a-71a8-47c1-9160-1b35c83e22a6","Type":"ContainerStarted","Data":"77450b60642c06305cf17147090d3ee2e5f562897e4daf37789e613e80dfebb9"}
Nov 02 01:30:08.794776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.794103    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" event={"ID":"d68e7f7a-71a8-47c1-9160-1b35c83e22a6","Type":"ContainerStarted","Data":"3ac4d6305662d234909f52f6a75bdee03563f02171b697aa3ad2fda4b0d5c940"}
Nov 02 01:30:08.804252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.799171    4851 generic.go:356] "Generic (PLEG): container finished" podID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerID="ce62c4084577514eb303da2af844c534758066e8a26f285eaca0f0f0620346fc" exitCode=60
Nov 02 01:30:08.804252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.799277    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" event={"ID":"ac436667-7100-41f1-b3ca-1bc1029f3266","Type":"ContainerDied","Data":"ce62c4084577514eb303da2af844c534758066e8a26f285eaca0f0f0620346fc"}
Nov 02 01:30:08.804252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.799755    4851 scope.go:122] "RemoveContainer" containerID="ce62c4084577514eb303da2af844c534758066e8a26f285eaca0f0f0620346fc"
Nov 02 01:30:08.817635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.816940    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6601-7d895f" podStartSLOduration=3.816920419 podStartE2EDuration="3.816920419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:08.778130887 +0000 UTC m=+914.083063714" watchObservedRunningTime="2026-11-02 01:30:08.816920419 +0000 UTC m=+914.121853247"
Nov 02 01:30:08.827562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.826819    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"]
Nov 02 01:30:08.845688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.842456    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" podStartSLOduration=8.842438448 podStartE2EDuration="8.842438448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:08.838552776 +0000 UTC m=+914.143485603" watchObservedRunningTime="2026-11-02 01:30:08.842438448 +0000 UTC m=+914.147371275"
Nov 02 01:30:08.851990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.851962    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:08.853328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.852657    4851 generic.go:356] "Generic (PLEG): container finished" podID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerID="74e9d31a3a71c7e149f218b948778dba616192f477391e8ac35517717b043e07" exitCode=60
Nov 02 01:30:08.853328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.853154    4851 scope.go:122] "RemoveContainer" containerID="74e9d31a3a71c7e149f218b948778dba616192f477391e8ac35517717b043e07"
Nov 02 01:30:08.861934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.859140    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"]
Nov 02 01:30:08.861934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.859231    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s" event={"ID":"f0aff15e-598c-4f9b-955d-d183f7c7dd38","Type":"ContainerStarted","Data":"4c8e9f7b8a8a20b0c5620f8889e78aa1a501283ddbace726cd0e244dd46aedb0"}
Nov 02 01:30:08.861934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.859250    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:08.861934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.859268    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s" event={"ID":"f0aff15e-598c-4f9b-955d-d183f7c7dd38","Type":"ContainerStarted","Data":"ba9dd1e7c0b65913b658065d29c8a88b35fe24d57a066d04076ba4c3566d009d"}
Nov 02 01:30:08.861934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.859309    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" event={"ID":"27fb8c50-9ff8-4be1-b961-6f2474b48804","Type":"ContainerDied","Data":"74e9d31a3a71c7e149f218b948778dba616192f477391e8ac35517717b043e07"}
Nov 02 01:30:08.883813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.883786    4851 scope.go:122] "RemoveContainer" containerID="1b703c7fa04ce3f2dc76e189082dd314fdaff2a85dba108fbb9099cf06818026"
Nov 02 01:30:08.982107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.942092    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2f21c877-openshift-default-dockercfg-rx7j7\""
Nov 02 01:30:08.982107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.981806    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50473a1b-e314-4f06-90b3-558dad6dc6cf-istiod-ca-cert\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:08.982107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:08.981920    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-socket\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.001700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.001659    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-data\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.002082 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.002054    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-credential-socket\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.002513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.002487    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvtb9\" (UniqueName: \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-kube-api-access-lvtb9\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.002718 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.002689    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-token\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.002925 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.002898    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-certs\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.003100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.003080    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-envoy\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.003219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.003204    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-podinfo\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.126929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.126833    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr" event={"ID":"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905","Type":"ContainerStarted","Data":"3fe5cfcdd3f619149bb6ced27616e826ff7c0ab982a0feb4c3e5ca5cdb0cfd44"}
Nov 02 01:30:09.188650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.129958    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-socket\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.188852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.188649    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp" event={"ID":"6b27e232-7a42-4130-b455-4973b6627bd5","Type":"ContainerStarted","Data":"1b703c7fa04ce3f2dc76e189082dd314fdaff2a85dba108fbb9099cf06818026"}
Nov 02 01:30:09.188852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.188779    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-data\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.188984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.130502    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-socket\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.188984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.188907    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-credential-socket\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.189315    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-credential-socket\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.189662    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-data\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.190733    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvtb9\" (UniqueName: \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-kube-api-access-lvtb9\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.190792    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-token\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.190857    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-certs\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.190921    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-envoy\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.190941    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-podinfo\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.191062    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50473a1b-e314-4f06-90b3-558dad6dc6cf-istiod-ca-cert\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.193049    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-certs\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.198212    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50473a1b-e314-4f06-90b3-558dad6dc6cf-istiod-ca-cert\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.341229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.340618    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-token\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.390359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.384083    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-envoy\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.394844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.394067    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvtb9\" (UniqueName: \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-kube-api-access-lvtb9\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.394844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.394415    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-podinfo\") pod \"e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") " pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.405217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.405137    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:09.473880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.473851    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:30:09.505675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.504004    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:09.537353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.522838    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e629ab-dc91-41b3-aa5e-b2a1724f7a00-kube-api-access-2n6pd\" (UniqueName: \"kubernetes.io/projected/e5e629ab-dc91-41b3-aa5e-b2a1724f7a00-kube-api-access-2n6pd\") pod \"e5e629ab-dc91-41b3-aa5e-b2a1724f7a00\" (UID: \"e5e629ab-dc91-41b3-aa5e-b2a1724f7a00\") "
Nov 02 01:30:09.567972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.567919    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e629ab-dc91-41b3-aa5e-b2a1724f7a00-kube-api-access-2n6pd" pod "e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" (UID: "e5e629ab-dc91-41b3-aa5e-b2a1724f7a00"). InnerVolumeSpecName "kube-api-access-2n6pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:09.621052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.613920    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:30:09.621052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.620864    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:09.636923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.635723    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:30:09.638732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:09.638669    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n6pd\" (UniqueName: \"kubernetes.io/projected/e5e629ab-dc91-41b3-aa5e-b2a1724f7a00-kube-api-access-2n6pd\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:10.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.005765    4851 generic.go:356] "Generic (PLEG): container finished" podID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerID="77450b60642c06305cf17147090d3ee2e5f562897e4daf37789e613e80dfebb9" exitCode=60
Nov 02 01:30:10.007655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.006344    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" event={"ID":"d68e7f7a-71a8-47c1-9160-1b35c83e22a6","Type":"ContainerDied","Data":"77450b60642c06305cf17147090d3ee2e5f562897e4daf37789e613e80dfebb9"}
Nov 02 01:30:10.007655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.006657    4851 scope.go:122] "RemoveContainer" containerID="77450b60642c06305cf17147090d3ee2e5f562897e4daf37789e613e80dfebb9"
Nov 02 01:30:10.018010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.017924    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:10.065746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.063550    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" event={"ID":"ac436667-7100-41f1-b3ca-1bc1029f3266","Type":"ContainerStarted","Data":"5db1aa7ff40a3b2598a512902e68601e96e5220160768f778ee23d3f73515b99"}
Nov 02 01:30:10.069072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.069029    4851 generic.go:356] "Generic (PLEG): container finished" podID="f0aff15e-598c-4f9b-955d-d183f7c7dd38" containerID="4c8e9f7b8a8a20b0c5620f8889e78aa1a501283ddbace726cd0e244dd46aedb0" exitCode=0
Nov 02 01:30:10.070781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.070697    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s" event={"ID":"f0aff15e-598c-4f9b-955d-d183f7c7dd38","Type":"ContainerDied","Data":"4c8e9f7b8a8a20b0c5620f8889e78aa1a501283ddbace726cd0e244dd46aedb0"}
Nov 02 01:30:10.086488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.086458    4851 generic.go:356] "Generic (PLEG): container finished" podID="3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905" containerID="c2ba7d93b5c2de35816f618cb8d4530995e4d7f70f3e37c03bc92b844afe9c5c" exitCode=0
Nov 02 01:30:10.086735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.086710    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr" event={"ID":"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905","Type":"ContainerDied","Data":"c2ba7d93b5c2de35816f618cb8d4530995e4d7f70f3e37c03bc92b844afe9c5c"}
Nov 02 01:30:10.093711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.093684    4851 generic.go:356] "Generic (PLEG): container finished" podID="6b27e232-7a42-4130-b455-4973b6627bd5" containerID="1b703c7fa04ce3f2dc76e189082dd314fdaff2a85dba108fbb9099cf06818026" exitCode=60
Nov 02 01:30:10.093976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.093955    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp" event={"ID":"6b27e232-7a42-4130-b455-4973b6627bd5","Type":"ContainerDied","Data":"1b703c7fa04ce3f2dc76e189082dd314fdaff2a85dba108fbb9099cf06818026"}
Nov 02 01:30:10.096790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.096010    4851 generic.go:356] "Generic (PLEG): container finished" podID="e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" containerID="2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214" exitCode=137
Nov 02 01:30:10.097425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.097126    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:30:10.101308 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.101276    4851 scope.go:122] "RemoveContainer" containerID="2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214"
Nov 02 01:30:10.189987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.189911    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" podStartSLOduration=10.189886332 podStartE2EDuration="10.189886332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:10.08405304 +0000 UTC m=+915.388985881" watchObservedRunningTime="2026-11-02 01:30:10.189886332 +0000 UTC m=+915.494819171"
Nov 02 01:30:10.243202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.243174    4851 scope.go:122] "RemoveContainer" containerID="2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214"
Nov 02 01:30:10.243938 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:10.243908    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214\": container with ID starting with 2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214 not found: ID does not exist" containerID="2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214"
Nov 02 01:30:10.244048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.243986    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214"} err="failed to get container status \"2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214\": rpc error: code = NotFound desc = could not find container \"2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214\": container with ID starting with 2ee382289cb4ed2ae284be79df4b85d61b86a8688f6badb01a8711a62fc1c214 not found: ID does not exist"
Nov 02 01:30:10.281690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.280157    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:10.339870 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:10.339601    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-b8edec\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" pod="opendatahub/test-curl-6601-b8edec"
Nov 02 01:30:10.616031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.615993    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:10.616231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.616056    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:10.678310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.675334    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:10.678310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.675420    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:10.884378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.883763    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:10.884378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.883868    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:10.884378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.883918    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:10.948568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.947217    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e629ab-dc91-41b3-aa5e-b2a1724f7a00" path="/var/lib/kubelet/pods/e5e629ab-dc91-41b3-aa5e-b2a1724f7a00/volumes"
Nov 02 01:30:10.948568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.947751    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6601-7d895f"]
Nov 02 01:30:10.948568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.947803    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6601-7d895f"]
Nov 02 01:30:10.948568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:10.947831    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"]
Nov 02 01:30:11.110659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.110568    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" event={"ID":"d68e7f7a-71a8-47c1-9160-1b35c83e22a6","Type":"ContainerStarted","Data":"726d9a20166c2c38f19700aab1ce2d124a326a7b9bc7188956e54c1ec0205615"}
Nov 02 01:30:11.116711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.116682    4851 generic.go:356] "Generic (PLEG): container finished" podID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerID="5db1aa7ff40a3b2598a512902e68601e96e5220160768f778ee23d3f73515b99" exitCode=60
Nov 02 01:30:11.116853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.116785    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" event={"ID":"ac436667-7100-41f1-b3ca-1bc1029f3266","Type":"ContainerDied","Data":"5db1aa7ff40a3b2598a512902e68601e96e5220160768f778ee23d3f73515b99"}
Nov 02 01:30:11.116853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.116837    4851 scope.go:122] "RemoveContainer" containerID="ce62c4084577514eb303da2af844c534758066e8a26f285eaca0f0f0620346fc"
Nov 02 01:30:11.117014 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.116997    4851 scope.go:122] "RemoveContainer" containerID="5db1aa7ff40a3b2598a512902e68601e96e5220160768f778ee23d3f73515b99"
Nov 02 01:30:11.117510 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:11.117481    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx_odh-ai-gateway-infra(ac436667-7100-41f1-b3ca-1bc1029f3266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266"
Nov 02 01:30:11.120094 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.120069    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s" event={"ID":"f0aff15e-598c-4f9b-955d-d183f7c7dd38","Type":"ContainerDied","Data":"ba9dd1e7c0b65913b658065d29c8a88b35fe24d57a066d04076ba4c3566d009d"}
Nov 02 01:30:11.120265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.120240    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba9dd1e7c0b65913b658065d29c8a88b35fe24d57a066d04076ba4c3566d009d"
Nov 02 01:30:11.122811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.122782    4851 generic.go:356] "Generic (PLEG): container finished" podID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerID="6cfaa37a003acc06e4cb40668ed25b00fb663b105e9be5474c2913768ca87bf4" exitCode=60
Nov 02 01:30:11.122905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.122871    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" event={"ID":"27fb8c50-9ff8-4be1-b961-6f2474b48804","Type":"ContainerDied","Data":"6cfaa37a003acc06e4cb40668ed25b00fb663b105e9be5474c2913768ca87bf4"}
Nov 02 01:30:11.123449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.123121    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:11.123449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.123356    4851 scope.go:122] "RemoveContainer" containerID="6cfaa37a003acc06e4cb40668ed25b00fb663b105e9be5474c2913768ca87bf4"
Nov 02 01:30:11.123842 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:11.123712    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg_odh-ai-gateway-infra(27fb8c50-9ff8-4be1-b961-6f2474b48804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804"
Nov 02 01:30:11.125497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.125477    4851 generic.go:356] "Generic (PLEG): container finished" podID="6b27e232-7a42-4130-b455-4973b6627bd5" containerID="08648fe8277bd74fca25516f64069df5a7a403909c4b9210e4b5411b283e3b07" exitCode=60
Nov 02 01:30:11.125579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.125558    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp" event={"ID":"6b27e232-7a42-4130-b455-4973b6627bd5","Type":"ContainerDied","Data":"08648fe8277bd74fca25516f64069df5a7a403909c4b9210e4b5411b283e3b07"}
Nov 02 01:30:11.125861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.125841    4851 scope.go:122] "RemoveContainer" containerID="08648fe8277bd74fca25516f64069df5a7a403909c4b9210e4b5411b283e3b07"
Nov 02 01:30:11.126177 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:11.126117    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp_odh-ai-gateway-infra(6b27e232-7a42-4130-b455-4973b6627bd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp" podUID="6b27e232-7a42-4130-b455-4973b6627bd5"
Nov 02 01:30:11.129247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.128759    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj" event={"ID":"50473a1b-e314-4f06-90b3-558dad6dc6cf","Type":"ContainerStarted","Data":"1a5bee9fb3c365b9e17f8f53f2ff1cec911adf63630334eeb17ef2ae499e6727"}
Nov 02 01:30:11.129560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.129535    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6601-7d895f" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" containerName="test-curl-6601-7d895f" containerID="cri-o://77d742256e00a342fbab27ee8596fcd49df78a49a073a306ac70357871161242" gracePeriod=2
Nov 02 01:30:11.133042 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:11.133009    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:11.145154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.145128    4851 scope.go:122] "RemoveContainer" containerID="74e9d31a3a71c7e149f218b948778dba616192f477391e8ac35517717b043e07"
Nov 02 01:30:11.176246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.176216    4851 scope.go:122] "RemoveContainer" containerID="1b703c7fa04ce3f2dc76e189082dd314fdaff2a85dba108fbb9099cf06818026"
Nov 02 01:30:11.259501 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:11.257305    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:11.311258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.308056    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-xks-ca\") pod \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") "
Nov 02 01:30:11.311258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.308435    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-openshift-ca\") pod \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") "
Nov 02 01:30:11.311258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.308499    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0aff15e-598c-4f9b-955d-d183f7c7dd38-kube-api-access-wxncq\" (UniqueName: \"kubernetes.io/projected/f0aff15e-598c-4f9b-955d-d183f7c7dd38-kube-api-access-wxncq\") pod \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\" (UID: \"f0aff15e-598c-4f9b-955d-d183f7c7dd38\") "
Nov 02 01:30:11.311613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.311589    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0aff15e-598c-4f9b-955d-d183f7c7dd38-kube-api-access-wxncq" pod "f0aff15e-598c-4f9b-955d-d183f7c7dd38" (UID: "f0aff15e-598c-4f9b-955d-d183f7c7dd38"). InnerVolumeSpecName "kube-api-access-wxncq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:11.312574 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.312549    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-openshift-ca" pod "f0aff15e-598c-4f9b-955d-d183f7c7dd38" (UID: "f0aff15e-598c-4f9b-955d-d183f7c7dd38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:11.313861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.313838    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-xks-ca" pod "f0aff15e-598c-4f9b-955d-d183f7c7dd38" (UID: "f0aff15e-598c-4f9b-955d-d183f7c7dd38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:30:11.362630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.362550    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:11.365546 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:11.365505    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:11.411039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.411001    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:11.411274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.411254    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxncq\" (UniqueName: \"kubernetes.io/projected/f0aff15e-598c-4f9b-955d-d183f7c7dd38-kube-api-access-wxncq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:11.411444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.411429    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0aff15e-598c-4f9b-955d-d183f7c7dd38-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:11.499328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.495684    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:11.528320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.514079    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-xks-ca\") pod \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") "
Nov 02 01:30:11.528874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.528840    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-kube-api-access-c5bfb\" (UniqueName: \"kubernetes.io/projected/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-kube-api-access-c5bfb\") pod \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") "
Nov 02 01:30:11.529188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.529167    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-openshift-ca\") pod \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\" (UID: \"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905\") "
Nov 02 01:30:11.639442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.594165    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-openshift-ca" pod "3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905" (UID: "3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:11.639442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.616473    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:11.639442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.616529    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:11.668780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.641602    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:11.668780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.656116    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:11.704313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.697652    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-kube-api-access-c5bfb" pod "3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905" (UID: "3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905"). InnerVolumeSpecName "kube-api-access-c5bfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:11.738537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.732413    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:11.738537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.732467    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:11.750467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.749741    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5bfb\" (UniqueName: \"kubernetes.io/projected/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-kube-api-access-c5bfb\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:11.779013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.766567    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-xks-ca" pod "3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905" (UID: "3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:30:11.788853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.788786    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:11.852177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:11.852143    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:12.218776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.218747    4851 scope.go:122] "RemoveContainer" containerID="5db1aa7ff40a3b2598a512902e68601e96e5220160768f778ee23d3f73515b99"
Nov 02 01:30:12.219505 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:12.219476    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx_odh-ai-gateway-infra(ac436667-7100-41f1-b3ca-1bc1029f3266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266"
Nov 02 01:30:12.222037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.222019    4851 scope.go:122] "RemoveContainer" containerID="6cfaa37a003acc06e4cb40668ed25b00fb663b105e9be5474c2913768ca87bf4"
Nov 02 01:30:12.223966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.223945    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr" event={"ID":"3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905","Type":"ContainerDied","Data":"3fe5cfcdd3f619149bb6ced27616e826ff7c0ab982a0feb4c3e5ca5cdb0cfd44"}
Nov 02 01:30:12.224094 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.224085    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fe5cfcdd3f619149bb6ced27616e826ff7c0ab982a0feb4c3e5ca5cdb0cfd44"
Nov 02 01:30:12.224203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.224195    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"
Nov 02 01:30:12.224277 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:12.224249    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg_odh-ai-gateway-infra(27fb8c50-9ff8-4be1-b961-6f2474b48804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804"
Nov 02 01:30:12.227914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.227890    4851 scope.go:122] "RemoveContainer" containerID="08648fe8277bd74fca25516f64069df5a7a403909c4b9210e4b5411b283e3b07"
Nov 02 01:30:12.231382 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:12.228953    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp_odh-ai-gateway-infra(6b27e232-7a42-4130-b455-4973b6627bd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp" podUID="6b27e232-7a42-4130-b455-4973b6627bd5"
Nov 02 01:30:12.285712 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:12.257265    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:12.315020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.314978    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj" event={"ID":"50473a1b-e314-4f06-90b3-558dad6dc6cf","Type":"ContainerStarted","Data":"5ce60ca53d5b22e64363c666dc12edb19941024dab4edd025dfb4f8f95d0361d"}
Nov 02 01:30:12.320079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.320036    4851 generic.go:356] "Generic (PLEG): container finished" podID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerID="726d9a20166c2c38f19700aab1ce2d124a326a7b9bc7188956e54c1ec0205615" exitCode=60
Nov 02 01:30:12.320261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.320216    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"
Nov 02 01:30:12.320758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.320729    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" event={"ID":"d68e7f7a-71a8-47c1-9160-1b35c83e22a6","Type":"ContainerDied","Data":"726d9a20166c2c38f19700aab1ce2d124a326a7b9bc7188956e54c1ec0205615"}
Nov 02 01:30:12.320870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.320808    4851 scope.go:122] "RemoveContainer" containerID="77450b60642c06305cf17147090d3ee2e5f562897e4daf37789e613e80dfebb9"
Nov 02 01:30:12.321343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.321171    4851 scope.go:122] "RemoveContainer" containerID="726d9a20166c2c38f19700aab1ce2d124a326a7b9bc7188956e54c1ec0205615"
Nov 02 01:30:12.321826 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:12.321785    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2_odh-ai-gateway-infra(d68e7f7a-71a8-47c1-9160-1b35c83e22a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6"
Nov 02 01:30:12.326226 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:12.326190    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:12.394356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.394277    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:30:12.396283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.396260    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:30:12.396819 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:12.396792    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:12.411515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.411479    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:30:12.423334 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:12.423245    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:12.429348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.429280    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj" podStartSLOduration=4.429265237 podStartE2EDuration="4.429265237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:12.420636177 +0000 UTC m=+917.725569003" watchObservedRunningTime="2026-11-02 01:30:12.429265237 +0000 UTC m=+917.734198064"
Nov 02 01:30:12.433154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.433126    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:30:12.475002 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:12.474909    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:12.507074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.507040    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:12.514465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:12.514433    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:12.550860 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:12.550824    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:13.332527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:13.332502    4851 generic.go:356] "Generic (PLEG): container finished" podID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" containerID="77d742256e00a342fbab27ee8596fcd49df78a49a073a306ac70357871161242" exitCode=137
Nov 02 01:30:13.335108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:13.335075    4851 scope.go:122] "RemoveContainer" containerID="726d9a20166c2c38f19700aab1ce2d124a326a7b9bc7188956e54c1ec0205615"
Nov 02 01:30:13.335412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:13.335393    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:13.335543 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:13.335523    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2_odh-ai-gateway-infra(d68e7f7a-71a8-47c1-9160-1b35c83e22a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6"
Nov 02 01:30:13.336504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:13.336486    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:30:13.363465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:13.363444    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:13.496683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:13.496653    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df26cce1-0a86-46a7-9892-1cd1cc76dd23-kube-api-access-g9hts\" (UniqueName: \"kubernetes.io/projected/df26cce1-0a86-46a7-9892-1cd1cc76dd23-kube-api-access-g9hts\") pod \"df26cce1-0a86-46a7-9892-1cd1cc76dd23\" (UID: \"df26cce1-0a86-46a7-9892-1cd1cc76dd23\") "
Nov 02 01:30:13.499373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:13.499347    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df26cce1-0a86-46a7-9892-1cd1cc76dd23-kube-api-access-g9hts" pod "df26cce1-0a86-46a7-9892-1cd1cc76dd23" (UID: "df26cce1-0a86-46a7-9892-1cd1cc76dd23"). InnerVolumeSpecName "kube-api-access-g9hts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:13.599378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:13.599348    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9hts\" (UniqueName: \"kubernetes.io/projected/df26cce1-0a86-46a7-9892-1cd1cc76dd23-kube-api-access-g9hts\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:13.648728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:13.648698    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:14.083468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.083435    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:14.244165 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.244131    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:14.359311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.359224    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:14.359311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.359256    4851 scope.go:122] "RemoveContainer" containerID="77d742256e00a342fbab27ee8596fcd49df78a49a073a306ac70357871161242"
Nov 02 01:30:14.370273 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:14.370188    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:14.381995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.381969    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:14.488764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.488732    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"]
Nov 02 01:30:14.489473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.489458    4851 cpu_manager.go:414] "containerMap: removing container" podUID="f0aff15e-598c-4f9b-955d-d183f7c7dd38" containerName="revoke-keys"
Nov 02 01:30:14.489541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.489474    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0aff15e-598c-4f9b-955d-d183f7c7dd38" containerName="revoke-keys"
Nov 02 01:30:14.489541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.489488    4851 cpu_manager.go:414] "containerMap: removing container" podUID="3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905" containerName="revoke-keys"
Nov 02 01:30:14.489541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.489493    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905" containerName="revoke-keys"
Nov 02 01:30:14.489541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.489520    4851 cpu_manager.go:414] "containerMap: removing container" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" containerName="test-curl-6601-7d895f"
Nov 02 01:30:14.489541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.489525    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" containerName="test-curl-6601-7d895f"
Nov 02 01:30:14.489703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.489633    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" containerName="test-curl-6601-7d895f"
Nov 02 01:30:14.489703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.489655    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0aff15e-598c-4f9b-955d-d183f7c7dd38" containerName="revoke-keys"
Nov 02 01:30:14.489703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.489663    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905" containerName="revoke-keys"
Nov 02 01:30:14.495006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.494983    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.498660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.498640    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:30:14.501615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.501595    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"]
Nov 02 01:30:14.514031 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:14.513965    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:14.640002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.639926    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j56cr\" (UniqueName: \"kubernetes.io/projected/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-kube-api-access-j56cr\") pod \"maas-api-revoke-sub-8655f22a-5l7ww\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.640176 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.640060    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8655f22a-5l7ww\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.640176 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.640116    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8655f22a-5l7ww\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.741723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.741688    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j56cr\" (UniqueName: \"kubernetes.io/projected/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-kube-api-access-j56cr\") pod \"maas-api-revoke-sub-8655f22a-5l7ww\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.741918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.741792    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8655f22a-5l7ww\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.741918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.741824    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8655f22a-5l7ww\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.742623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.742603    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8655f22a-5l7ww\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.744280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.744261    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8655f22a-5l7ww\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.754445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.754407    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j56cr\" (UniqueName: \"kubernetes.io/projected/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-kube-api-access-j56cr\") pod \"maas-api-revoke-sub-8655f22a-5l7ww\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.816567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.816544    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:14.851771 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:14.851736    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6601-7d895f\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" pod="opendatahub/test-curl-6601-7d895f"
Nov 02 01:30:14.852546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.852506    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df26cce1-0a86-46a7-9892-1cd1cc76dd23" path="/var/lib/kubelet/pods/df26cce1-0a86-46a7-9892-1cd1cc76dd23/volumes"
Nov 02 01:30:14.929219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:14.929186    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:15.125046 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:15.125018    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90aaf3f5_e6c3_40f9_bd39_c06c8240d012.slice/crio-f0a801fa425a8721509947b1d5894a474cf2368a12437c9b5e7d661b8adab3d5 WatchSource:0}: Error finding container f0a801fa425a8721509947b1d5894a474cf2368a12437c9b5e7d661b8adab3d5: Status 404 returned error can't find the container with id f0a801fa425a8721509947b1d5894a474cf2368a12437c9b5e7d661b8adab3d5
Nov 02 01:30:15.129148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.129120    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"]
Nov 02 01:30:15.328050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.327945    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:15.365504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.365474    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"]
Nov 02 01:30:15.371652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.371630    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:30:15.372301 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.372257    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww" event={"ID":"90aaf3f5-e6c3-40f9-bd39-c06c8240d012","Type":"ContainerStarted","Data":"f0a801fa425a8721509947b1d5894a474cf2368a12437c9b5e7d661b8adab3d5"}
Nov 02 01:30:15.380120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.380095    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-854ec26e-dockercfg-bjhcb\""
Nov 02 01:30:15.384540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.384516    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-854ec26e\""
Nov 02 01:30:15.385512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.385491    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"]
Nov 02 01:30:15.424858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.424820    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"]
Nov 02 01:30:15.429655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.429630    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:30:15.444776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.444750    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"]
Nov 02 01:30:15.499774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.489478    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:15.590358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.583554    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6fx\" (UniqueName: \"kubernetes.io/projected/489366b9-640d-4d92-95b5-ddb11c36f8be-kube-api-access-tm6fx\") pod \"payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn\" (UID: \"489366b9-640d-4d92-95b5-ddb11c36f8be\") " pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:30:15.590358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.583640    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4454f04-826e-45f4-af38-db0bbd64603c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2\" (UID: \"a4454f04-826e-45f4-af38-db0bbd64603c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:30:15.590358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.583682    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489366b9-640d-4d92-95b5-ddb11c36f8be-plugins-config-volume\") pod \"payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn\" (UID: \"489366b9-640d-4d92-95b5-ddb11c36f8be\") " pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:30:15.590358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.583852    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstqc\" (UniqueName: \"kubernetes.io/projected/a4454f04-826e-45f4-af38-db0bbd64603c-kube-api-access-gstqc\") pod \"payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2\" (UID: \"a4454f04-826e-45f4-af38-db0bbd64603c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:30:15.878214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.878108    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489366b9-640d-4d92-95b5-ddb11c36f8be-plugins-config-volume\") pod \"payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn\" (UID: \"489366b9-640d-4d92-95b5-ddb11c36f8be\") " pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:30:15.880336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.879217    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489366b9-640d-4d92-95b5-ddb11c36f8be-plugins-config-volume\") pod \"payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn\" (UID: \"489366b9-640d-4d92-95b5-ddb11c36f8be\") " pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:30:15.880336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.879694    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstqc\" (UniqueName: \"kubernetes.io/projected/a4454f04-826e-45f4-af38-db0bbd64603c-kube-api-access-gstqc\") pod \"payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2\" (UID: \"a4454f04-826e-45f4-af38-db0bbd64603c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:30:15.880576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.880558    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6fx\" (UniqueName: \"kubernetes.io/projected/489366b9-640d-4d92-95b5-ddb11c36f8be-kube-api-access-tm6fx\") pod \"payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn\" (UID: \"489366b9-640d-4d92-95b5-ddb11c36f8be\") " pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:30:15.880659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.880642    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4454f04-826e-45f4-af38-db0bbd64603c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2\" (UID: \"a4454f04-826e-45f4-af38-db0bbd64603c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:30:15.881368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.881348    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4454f04-826e-45f4-af38-db0bbd64603c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2\" (UID: \"a4454f04-826e-45f4-af38-db0bbd64603c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:30:15.892461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.892438    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstqc\" (UniqueName: \"kubernetes.io/projected/a4454f04-826e-45f4-af38-db0bbd64603c-kube-api-access-gstqc\") pod \"payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2\" (UID: \"a4454f04-826e-45f4-af38-db0bbd64603c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:30:15.892733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.892715    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6fx\" (UniqueName: \"kubernetes.io/projected/489366b9-640d-4d92-95b5-ddb11c36f8be-kube-api-access-tm6fx\") pod \"payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn\" (UID: \"489366b9-640d-4d92-95b5-ddb11c36f8be\") " pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:30:15.991947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:15.991921    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:30:16.009334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.009270    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:16.130841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.129581    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:30:16.383433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.383365    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:16.384359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.384335    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww" event={"ID":"90aaf3f5-e6c3-40f9-bd39-c06c8240d012","Type":"ContainerStarted","Data":"f8767c1a789b9eb72fc6cdc433e916972d0741e87d8916105f08d46fdd93bdce"}
Nov 02 01:30:16.395121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.395102    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"]
Nov 02 01:30:16.397914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:16.397881    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4454f04_826e_45f4_af38_db0bbd64603c.slice/crio-eb62bbb27c3adb91489716b553ece01273359d9d0d64a177e5e60d3fe37e961c WatchSource:0}: Error finding container eb62bbb27c3adb91489716b553ece01273359d9d0d64a177e5e60d3fe37e961c: Status 404 returned error can't find the container with id eb62bbb27c3adb91489716b553ece01273359d9d0d64a177e5e60d3fe37e961c
Nov 02 01:30:16.402693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.402670    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"]
Nov 02 01:30:16.408971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.408953    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:30:16.415319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.414664    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2f21c877\""
Nov 02 01:30:16.436950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.434926    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww" podStartSLOduration=2.434907779 podStartE2EDuration="2.434907779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:16.41348243 +0000 UTC m=+921.718415259" watchObservedRunningTime="2026-11-02 01:30:16.434907779 +0000 UTC m=+921.739840606"
Nov 02 01:30:16.436950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.435567    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"]
Nov 02 01:30:16.466712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.466674    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"]
Nov 02 01:30:16.498245 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:16.498211    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489366b9_640d_4d92_95b5_ddb11c36f8be.slice/crio-da099accfb07f89625fb279f781a271ce7b1178f204d82b0e0fd6b4121c86b62 WatchSource:0}: Error finding container da099accfb07f89625fb279f781a271ce7b1178f204d82b0e0fd6b4121c86b62: Status 404 returned error can't find the container with id da099accfb07f89625fb279f781a271ce7b1178f204d82b0e0fd6b4121c86b62
Nov 02 01:30:16.500164 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.500133    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwgxw\" (UniqueName: \"kubernetes.io/projected/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-kube-api-access-cwgxw\") pod \"payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf\" (UID: \"e92bea82-bd88-49db-a9cb-c0bb3e51cd42\") " pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:30:16.508690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.506949    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-plugins-config-volume\") pod \"payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf\" (UID: \"e92bea82-bd88-49db-a9cb-c0bb3e51cd42\") " pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:30:16.543318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.524890    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:16.612418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.611995    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwgxw\" (UniqueName: \"kubernetes.io/projected/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-kube-api-access-cwgxw\") pod \"payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf\" (UID: \"e92bea82-bd88-49db-a9cb-c0bb3e51cd42\") " pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:30:16.624629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.624560    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-plugins-config-volume\") pod \"payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf\" (UID: \"e92bea82-bd88-49db-a9cb-c0bb3e51cd42\") " pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:30:16.624884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.612968    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-plugins-config-volume\") pod \"payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf\" (UID: \"e92bea82-bd88-49db-a9cb-c0bb3e51cd42\") " pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:30:16.645658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.645587    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwgxw\" (UniqueName: \"kubernetes.io/projected/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-kube-api-access-cwgxw\") pod \"payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf\" (UID: \"e92bea82-bd88-49db-a9cb-c0bb3e51cd42\") " pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:30:16.752994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:16.752469    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:30:17.094349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.093570    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:17.198921 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:17.198879    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92bea82_bd88_49db_a9cb_c0bb3e51cd42.slice/crio-dd5c608340810cc2042abb448724a1c0db06c997f3f4046be376feec80a9b276 WatchSource:0}: Error finding container dd5c608340810cc2042abb448724a1c0db06c997f3f4046be376feec80a9b276: Status 404 returned error can't find the container with id dd5c608340810cc2042abb448724a1c0db06c997f3f4046be376feec80a9b276
Nov 02 01:30:17.201609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.201582    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"]
Nov 02 01:30:17.397154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.397123    4851 generic.go:356] "Generic (PLEG): container finished" podID="90aaf3f5-e6c3-40f9-bd39-c06c8240d012" containerID="f8767c1a789b9eb72fc6cdc433e916972d0741e87d8916105f08d46fdd93bdce" exitCode=0
Nov 02 01:30:17.397669 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.397202    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww" event={"ID":"90aaf3f5-e6c3-40f9-bd39-c06c8240d012","Type":"ContainerDied","Data":"f8767c1a789b9eb72fc6cdc433e916972d0741e87d8916105f08d46fdd93bdce"}
Nov 02 01:30:17.399050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.399021    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn" event={"ID":"489366b9-640d-4d92-95b5-ddb11c36f8be","Type":"ContainerStarted","Data":"a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4"}
Nov 02 01:30:17.399172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.399074    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn" event={"ID":"489366b9-640d-4d92-95b5-ddb11c36f8be","Type":"ContainerStarted","Data":"da099accfb07f89625fb279f781a271ce7b1178f204d82b0e0fd6b4121c86b62"}
Nov 02 01:30:17.399172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.399103    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:30:17.401061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.401027    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2" event={"ID":"a4454f04-826e-45f4-af38-db0bbd64603c","Type":"ContainerStarted","Data":"dcce99b3f1f3801d9f929d81c3c456bf14ade44f3d70df4be10501ec19fcf365"}
Nov 02 01:30:17.401165 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.401087    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2" event={"ID":"a4454f04-826e-45f4-af38-db0bbd64603c","Type":"ContainerStarted","Data":"eb62bbb27c3adb91489716b553ece01273359d9d0d64a177e5e60d3fe37e961c"}
Nov 02 01:30:17.401165 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.401121    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:30:17.402971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.402948    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf" event={"ID":"e92bea82-bd88-49db-a9cb-c0bb3e51cd42","Type":"ContainerStarted","Data":"55b3ea302f6e8487694c36b97616e0b1c17db9f009e39e0dc58ec08cf22026a3"}
Nov 02 01:30:17.403069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.402995    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:30:17.403069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.403017    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf" event={"ID":"e92bea82-bd88-49db-a9cb-c0bb3e51cd42","Type":"ContainerStarted","Data":"dd5c608340810cc2042abb448724a1c0db06c997f3f4046be376feec80a9b276"}
Nov 02 01:30:17.444229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.444182    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf" podStartSLOduration=1.444170092 podStartE2EDuration="1.444170092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:17.437911285 +0000 UTC m=+922.742844113" watchObservedRunningTime="2026-11-02 01:30:17.444170092 +0000 UTC m=+922.749102942"
Nov 02 01:30:17.466658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.466565    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn" podStartSLOduration=2.466551207 podStartE2EDuration="2.466551207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:17.45786259 +0000 UTC m=+922.762795428" watchObservedRunningTime="2026-11-02 01:30:17.466551207 +0000 UTC m=+922.771484033"
Nov 02 01:30:17.484007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.483954    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2" podStartSLOduration=2.483939749 podStartE2EDuration="2.483939749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:17.48134437 +0000 UTC m=+922.786277198" watchObservedRunningTime="2026-11-02 01:30:17.483939749 +0000 UTC m=+922.788872574"
Nov 02 01:30:17.657740 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:17.657647    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:30:17.772360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.765730    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:17.908317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:17.908205    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:18.066394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.066160    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:18.249025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.248991    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:18.420625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.420591    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:18.471046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.471014    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"]
Nov 02 01:30:18.484591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.484562    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.485479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.485446    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"]
Nov 02 01:30:18.488555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.488534    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6c45ad5e-openshift-default-dockercfg-zst7s\""
Nov 02 01:30:18.527545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.527510    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-credential-socket\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.527710 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.527680    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istiod-ca-cert\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.527787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.527749    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-socket\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.527835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.527822    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-podinfo\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.527888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.527872    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxkx8\" (UniqueName: \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-kube-api-access-jxkx8\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.527938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.527919    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-token\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.528130 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.528107    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-certs\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.528265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.528170    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-envoy\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.528265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.528203    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-data\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.571972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.571934    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:18.690365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.667853    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-certs\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.693584    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-envoy\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.693663    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-data\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.693754    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-credential-socket\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.693913    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istiod-ca-cert\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.693995    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-socket\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.694025    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-podinfo\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.694084    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxkx8\" (UniqueName: \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-kube-api-access-jxkx8\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.694143    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-token\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.690558    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-certs\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.695613    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istiod-ca-cert\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.695857    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-data\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.696098    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-credential-socket\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.704779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.700589    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-socket\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.777435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.764600    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:18.921198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.918723    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxkx8\" (UniqueName: \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-kube-api-access-jxkx8\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.937214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.937139    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-envoy\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:18.937936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:18.937911    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-podinfo\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:19.017637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.016407    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-token\") pod \"e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") " pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:19.042738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.040505    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:19.106318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.106186    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:19.117132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.117108    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-xks-ca\") pod \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") "
Nov 02 01:30:19.117233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.117208    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-openshift-ca\") pod \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") "
Nov 02 01:30:19.117537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.117518    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-kube-api-access-j56cr\" (UniqueName: \"kubernetes.io/projected/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-kube-api-access-j56cr\") pod \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\" (UID: \"90aaf3f5-e6c3-40f9-bd39-c06c8240d012\") "
Nov 02 01:30:19.117644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.117620    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-openshift-ca" pod "90aaf3f5-e6c3-40f9-bd39-c06c8240d012" (UID: "90aaf3f5-e6c3-40f9-bd39-c06c8240d012"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:19.118799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.118779    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:19.119479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.119452    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-xks-ca" pod "90aaf3f5-e6c3-40f9-bd39-c06c8240d012" (UID: "90aaf3f5-e6c3-40f9-bd39-c06c8240d012"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:30:19.119960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.119939    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-kube-api-access-j56cr" pod "90aaf3f5-e6c3-40f9-bd39-c06c8240d012" (UID: "90aaf3f5-e6c3-40f9-bd39-c06c8240d012"). InnerVolumeSpecName "kube-api-access-j56cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:19.220100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.220068    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:19.220100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.220097    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j56cr\" (UniqueName: \"kubernetes.io/projected/90aaf3f5-e6c3-40f9-bd39-c06c8240d012-kube-api-access-j56cr\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:19.330571 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.330472    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:19.441544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.441498    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"]
Nov 02 01:30:19.467853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.467819    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww" event={"ID":"90aaf3f5-e6c3-40f9-bd39-c06c8240d012","Type":"ContainerDied","Data":"f0a801fa425a8721509947b1d5894a474cf2368a12437c9b5e7d661b8adab3d5"}
Nov 02 01:30:19.467981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.467897    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a801fa425a8721509947b1d5894a474cf2368a12437c9b5e7d661b8adab3d5"
Nov 02 01:30:19.468376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.468353    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"
Nov 02 01:30:19.470023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.469996    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:19.470178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.470162    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:19.470265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:19.470251    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:20.233959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:20.233926    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:20.479584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:20.479548    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg" event={"ID":"9563448c-19ba-4eb3-9ede-9429e5e6a9f2","Type":"ContainerStarted","Data":"03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee"}
Nov 02 01:30:20.480032 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:20.479600    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg" event={"ID":"9563448c-19ba-4eb3-9ede-9429e5e6a9f2","Type":"ContainerStarted","Data":"967913650a6b9e2f9e9b5fd43bee3a8e400d6629c0345498d2829256f48b99cd"}
Nov 02 01:30:20.505797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:20.505701    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg" podStartSLOduration=2.505688497 podStartE2EDuration="2.505688497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:20.504178021 +0000 UTC m=+925.809110866" watchObservedRunningTime="2026-11-02 01:30:20.505688497 +0000 UTC m=+925.810621323"
Nov 02 01:30:21.106767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:21.106726    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:21.112814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:21.112788    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:21.203123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:21.203086    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:21.488160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:21.488131    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:21.489186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:21.489171    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:30:21.839734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:21.839662    4851 scope.go:122] "RemoveContainer" containerID="5db1aa7ff40a3b2598a512902e68601e96e5220160768f778ee23d3f73515b99"
Nov 02 01:30:22.273798 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:22.273766    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:22.488268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:22.488190    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:22.763343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:22.758533    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" event={"ID":"ac436667-7100-41f1-b3ca-1bc1029f3266","Type":"ContainerStarted","Data":"66d817c6ba48b83adbb737aae480425499ea331f293d3ca1060101b6f0313874"}
Nov 02 01:30:23.521121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.521076    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:23.664109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.663895    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:23.778322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.778230    4851 generic.go:356] "Generic (PLEG): container finished" podID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerID="66d817c6ba48b83adbb737aae480425499ea331f293d3ca1060101b6f0313874" exitCode=60
Nov 02 01:30:23.778475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.778343    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" event={"ID":"ac436667-7100-41f1-b3ca-1bc1029f3266","Type":"ContainerDied","Data":"66d817c6ba48b83adbb737aae480425499ea331f293d3ca1060101b6f0313874"}
Nov 02 01:30:23.778475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.778403    4851 scope.go:122] "RemoveContainer" containerID="5db1aa7ff40a3b2598a512902e68601e96e5220160768f778ee23d3f73515b99"
Nov 02 01:30:23.779093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.779069    4851 scope.go:122] "RemoveContainer" containerID="66d817c6ba48b83adbb737aae480425499ea331f293d3ca1060101b6f0313874"
Nov 02 01:30:23.779782 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:23.779563    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx_odh-ai-gateway-infra(ac436667-7100-41f1-b3ca-1bc1029f3266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266"
Nov 02 01:30:23.803344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.803260    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:23.807945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.807907    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"]
Nov 02 01:30:23.839226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.839202    4851 scope.go:122] "RemoveContainer" containerID="6cfaa37a003acc06e4cb40668ed25b00fb663b105e9be5474c2913768ca87bf4"
Nov 02 01:30:23.839447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.839430    4851 scope.go:122] "RemoveContainer" containerID="08648fe8277bd74fca25516f64069df5a7a403909c4b9210e4b5411b283e3b07"
Nov 02 01:30:23.839575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:23.839558    4851 scope.go:122] "RemoveContainer" containerID="726d9a20166c2c38f19700aab1ce2d124a326a7b9bc7188956e54c1ec0205615"
Nov 02 01:30:24.132341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.132285    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:24.262228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.261920    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"]
Nov 02 01:30:24.263140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.263080    4851 cpu_manager.go:414] "containerMap: removing container" podUID="90aaf3f5-e6c3-40f9-bd39-c06c8240d012" containerName="revoke-keys"
Nov 02 01:30:24.263140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.263105    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90aaf3f5-e6c3-40f9-bd39-c06c8240d012" containerName="revoke-keys"
Nov 02 01:30:24.287324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.263377    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="90aaf3f5-e6c3-40f9-bd39-c06c8240d012" containerName="revoke-keys"
Nov 02 01:30:24.305372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.301371    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"]
Nov 02 01:30:24.305372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.301548    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.438326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.388895    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:24.592697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.519724    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.592697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.519789    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.592697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.519864    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.592697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.519894    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hpr9\" (UniqueName: \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-kube-api-access-6hpr9\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.592697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.519915    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.592697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.520101    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.592697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.520155    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.592697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.520213    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.592697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.520371    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.722405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.710778    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.722711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.722669    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hpr9\" (UniqueName: \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-kube-api-access-6hpr9\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.722868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.722853    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.723196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.723175    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.729432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.729403    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.729679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.729660    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.814940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.729940    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.842347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.826605    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.842347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.826673    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.842347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.816224    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:24.842347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.734010    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.842347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.827348    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.842347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.815787    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.842347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.828061    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.842347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.828219    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.948863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.948832    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.959108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.958666    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hpr9\" (UniqueName: \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-kube-api-access-6hpr9\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.959346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.959321    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.959829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.959807    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:24.978210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:24.975744    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:25.072963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.072931    4851 generic.go:356] "Generic (PLEG): container finished" podID="6b27e232-7a42-4130-b455-4973b6627bd5" containerID="26737f40ff736176664324146bb81e235d3e43b791e31b9e9a9c05fef2a4bf70" exitCode=60
Nov 02 01:30:25.084592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.084561    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" event={"ID":"27fb8c50-9ff8-4be1-b961-6f2474b48804","Type":"ContainerStarted","Data":"2ec83072daeed7ffcfa77af89b7dec5a4d0bd89d4c1f6d2519c1f9c9e5f888b1"}
Nov 02 01:30:25.085950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.084786    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp" event={"ID":"6b27e232-7a42-4130-b455-4973b6627bd5","Type":"ContainerDied","Data":"26737f40ff736176664324146bb81e235d3e43b791e31b9e9a9c05fef2a4bf70"}
Nov 02 01:30:25.085950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.084822    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" event={"ID":"d68e7f7a-71a8-47c1-9160-1b35c83e22a6","Type":"ContainerStarted","Data":"cfaa53cf8b82a1213fe821068476a311f2e7bcb8ddf52a0d72a4d711095d175a"}
Nov 02 01:30:25.085950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.084841    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"]
Nov 02 01:30:25.085950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.085755    4851 scope.go:122] "RemoveContainer" containerID="08648fe8277bd74fca25516f64069df5a7a403909c4b9210e4b5411b283e3b07"
Nov 02 01:30:25.085950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.085871    4851 scope.go:122] "RemoveContainer" containerID="26737f40ff736176664324146bb81e235d3e43b791e31b9e9a9c05fef2a4bf70"
Nov 02 01:30:25.086570 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:25.086480    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp_odh-ai-gateway-infra(6b27e232-7a42-4130-b455-4973b6627bd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp" podUID="6b27e232-7a42-4130-b455-4973b6627bd5"
Nov 02 01:30:25.087091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.087065    4851 scope.go:122] "RemoveContainer" containerID="cfaa53cf8b82a1213fe821068476a311f2e7bcb8ddf52a0d72a4d711095d175a"
Nov 02 01:30:25.087470 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:25.087445    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2_odh-ai-gateway-infra(d68e7f7a-71a8-47c1-9160-1b35c83e22a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6"
Nov 02 01:30:25.093196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.093166    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.095685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.095663    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"]
Nov 02 01:30:25.096999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.096971    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:30:25.302367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.274962    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aba21e8-6z8sj\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.302367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.275031    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfvxk\" (UniqueName: \"kubernetes.io/projected/209f7830-55fc-48b9-8347-5bd2c24ce5e2-kube-api-access-sfvxk\") pod \"maas-api-revoke-sub-9aba21e8-6z8sj\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.302367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.275234    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aba21e8-6z8sj\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.302367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.276976    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:25.397248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.397204    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aba21e8-6z8sj\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.418501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.418465    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfvxk\" (UniqueName: \"kubernetes.io/projected/209f7830-55fc-48b9-8347-5bd2c24ce5e2-kube-api-access-sfvxk\") pod \"maas-api-revoke-sub-9aba21e8-6z8sj\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.426039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.425833    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aba21e8-6z8sj\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.426457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.426435    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aba21e8-6z8sj\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.427710 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.427535    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aba21e8-6z8sj\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.490998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.490966    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfvxk\" (UniqueName: \"kubernetes.io/projected/209f7830-55fc-48b9-8347-5bd2c24ce5e2-kube-api-access-sfvxk\") pod \"maas-api-revoke-sub-9aba21e8-6z8sj\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.503073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.502600    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"
Nov 02 01:30:25.559776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.529761    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac436667-7100-41f1-b3ca-1bc1029f3266-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac436667-7100-41f1-b3ca-1bc1029f3266-maas-api-service-ca\") pod \"ac436667-7100-41f1-b3ca-1bc1029f3266\" (UID: \"ac436667-7100-41f1-b3ca-1bc1029f3266\") "
Nov 02 01:30:25.559776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.529851    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac436667-7100-41f1-b3ca-1bc1029f3266-kube-api-access-pjxbx\" (UniqueName: \"kubernetes.io/projected/ac436667-7100-41f1-b3ca-1bc1029f3266-kube-api-access-pjxbx\") pod \"ac436667-7100-41f1-b3ca-1bc1029f3266\" (UID: \"ac436667-7100-41f1-b3ca-1bc1029f3266\") "
Nov 02 01:30:25.559776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.532757    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac436667-7100-41f1-b3ca-1bc1029f3266-maas-api-service-ca" pod "ac436667-7100-41f1-b3ca-1bc1029f3266" (UID: "ac436667-7100-41f1-b3ca-1bc1029f3266"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:25.559776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.557504    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac436667-7100-41f1-b3ca-1bc1029f3266-kube-api-access-pjxbx" pod "ac436667-7100-41f1-b3ca-1bc1029f3266" (UID: "ac436667-7100-41f1-b3ca-1bc1029f3266"). InnerVolumeSpecName "kube-api-access-pjxbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:25.632558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.632526    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac436667-7100-41f1-b3ca-1bc1029f3266-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:25.632558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.632561    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjxbx\" (UniqueName: \"kubernetes.io/projected/ac436667-7100-41f1-b3ca-1bc1029f3266-kube-api-access-pjxbx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:25.697075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.697040    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"]
Nov 02 01:30:25.698774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.698749    4851 cpu_manager.go:414] "containerMap: removing container" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerName="cleanup"
Nov 02 01:30:25.698946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.698924    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerName="cleanup"
Nov 02 01:30:25.699045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.698974    4851 cpu_manager.go:414] "containerMap: removing container" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerName="cleanup"
Nov 02 01:30:25.699045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.698982    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerName="cleanup"
Nov 02 01:30:25.699346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.699327    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerName="cleanup"
Nov 02 01:30:25.699482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.699465    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerName="cleanup"
Nov 02 01:30:25.704271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.704251    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.708483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.708455    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-616e3ed7-openshift-default-dockercfg-85m2g\""
Nov 02 01:30:25.708861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.708836    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"]
Nov 02 01:30:25.731547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.728083    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:25.734241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.734208    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-socket\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.734379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.734252    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-token\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.734439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.734403    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-credential-socket\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.734439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.734430    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brl9f\" (UniqueName: \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-kube-api-access-brl9f\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.734537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.734454    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-certs\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.734537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.734482    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-podinfo\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.734649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.734567    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-data\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.734779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.734765    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83e32e09-9992-4e33-906c-cb088eb20ddc-istiod-ca-cert\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.734936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.734910    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-envoy\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.761461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.761435    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.848096    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83e32e09-9992-4e33-906c-cb088eb20ddc-istiod-ca-cert\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.848327    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-envoy\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.848508    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-socket\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.848532    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-token\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.848586    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-credential-socket\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.848621    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brl9f\" (UniqueName: \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-kube-api-access-brl9f\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.848656    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-certs\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.848683    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-podinfo\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.848819    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-data\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.849355    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-data\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.850026    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83e32e09-9992-4e33-906c-cb088eb20ddc-istiod-ca-cert\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.850687    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-credential-socket\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.850953    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-socket\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.857362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.851640    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-certs\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.903807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.903140    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brl9f\" (UniqueName: \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-kube-api-access-brl9f\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.918557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.906232    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-token\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.950558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.947314    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:25.950558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.947407    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:25.950558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.947452    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:25.960706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.960113    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-podinfo\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:25.963489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.963465    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-envoy\") pod \"e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") " pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:26.001094 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:25.999518    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:26.001094 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.000639    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"]
Nov 02 01:30:26.032622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.032591    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:26.114048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.102424    4851 generic.go:356] "Generic (PLEG): container finished" podID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerID="2ec83072daeed7ffcfa77af89b7dec5a4d0bd89d4c1f6d2519c1f9c9e5f888b1" exitCode=60
Nov 02 01:30:26.114048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.102471    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" event={"ID":"27fb8c50-9ff8-4be1-b961-6f2474b48804","Type":"ContainerDied","Data":"2ec83072daeed7ffcfa77af89b7dec5a4d0bd89d4c1f6d2519c1f9c9e5f888b1"}
Nov 02 01:30:26.114048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.102530    4851 scope.go:122] "RemoveContainer" containerID="6cfaa37a003acc06e4cb40668ed25b00fb663b105e9be5474c2913768ca87bf4"
Nov 02 01:30:26.114048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.103062    4851 scope.go:122] "RemoveContainer" containerID="2ec83072daeed7ffcfa77af89b7dec5a4d0bd89d4c1f6d2519c1f9c9e5f888b1"
Nov 02 01:30:26.114048 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:26.103552    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg_odh-ai-gateway-infra(27fb8c50-9ff8-4be1-b961-6f2474b48804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804"
Nov 02 01:30:26.114048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.110833    4851 generic.go:356] "Generic (PLEG): container finished" podID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerID="cfaa53cf8b82a1213fe821068476a311f2e7bcb8ddf52a0d72a4d711095d175a" exitCode=60
Nov 02 01:30:26.114048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.110951    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" event={"ID":"d68e7f7a-71a8-47c1-9160-1b35c83e22a6","Type":"ContainerDied","Data":"cfaa53cf8b82a1213fe821068476a311f2e7bcb8ddf52a0d72a4d711095d175a"}
Nov 02 01:30:26.114048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.111383    4851 scope.go:122] "RemoveContainer" containerID="cfaa53cf8b82a1213fe821068476a311f2e7bcb8ddf52a0d72a4d711095d175a"
Nov 02 01:30:26.114048 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:26.111747    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2_odh-ai-gateway-infra(d68e7f7a-71a8-47c1-9160-1b35c83e22a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6"
Nov 02 01:30:26.118099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.118067    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx" event={"ID":"ac436667-7100-41f1-b3ca-1bc1029f3266","Type":"ContainerDied","Data":"6c101e61a7377aff41037a05781ab83293c95d24b36014e6890a1b77eca687da"}
Nov 02 01:30:26.118263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.118227    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"
Nov 02 01:30:26.124465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.124438    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" event={"ID":"afc5860f-cfc3-4caf-844b-92dcbe2d91e4","Type":"ContainerStarted","Data":"0b77e6946154d2a8e5e8d36202f42428126704cd325c3784031f3ae22a2c8010"}
Nov 02 01:30:26.139816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.133794    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"]
Nov 02 01:30:26.139816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.139282    4851 scope.go:122] "RemoveContainer" containerID="726d9a20166c2c38f19700aab1ce2d124a326a7b9bc7188956e54c1ec0205615"
Nov 02 01:30:26.143976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.143953    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:26.322467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.322423    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"]
Nov 02 01:30:26.323984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.323954    4851 scope.go:122] "RemoveContainer" containerID="66d817c6ba48b83adbb737aae480425499ea331f293d3ca1060101b6f0313874"
Nov 02 01:30:26.372628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.372521    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"]
Nov 02 01:30:26.426183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.426133    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"]
Nov 02 01:30:26.435743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.434943    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-47dd19-29893050-nn4bx"]
Nov 02 01:30:26.445897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.445863    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"]
Nov 02 01:30:26.539929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.539894    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"
Nov 02 01:30:26.575206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.575169    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b27e232-7a42-4130-b455-4973b6627bd5-kube-api-access-mqxxw\" (UniqueName: \"kubernetes.io/projected/6b27e232-7a42-4130-b455-4973b6627bd5-kube-api-access-mqxxw\") pod \"6b27e232-7a42-4130-b455-4973b6627bd5\" (UID: \"6b27e232-7a42-4130-b455-4973b6627bd5\") "
Nov 02 01:30:26.575371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.575314    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b27e232-7a42-4130-b455-4973b6627bd5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b27e232-7a42-4130-b455-4973b6627bd5-maas-api-service-ca\") pod \"6b27e232-7a42-4130-b455-4973b6627bd5\" (UID: \"6b27e232-7a42-4130-b455-4973b6627bd5\") "
Nov 02 01:30:26.576919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.575648    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b27e232-7a42-4130-b455-4973b6627bd5-maas-api-service-ca" pod "6b27e232-7a42-4130-b455-4973b6627bd5" (UID: "6b27e232-7a42-4130-b455-4973b6627bd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:26.576919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.576160    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b27e232-7a42-4130-b455-4973b6627bd5-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:26.594519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.594487    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"]
Nov 02 01:30:26.654990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.648239    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b27e232-7a42-4130-b455-4973b6627bd5-kube-api-access-mqxxw" pod "6b27e232-7a42-4130-b455-4973b6627bd5" (UID: "6b27e232-7a42-4130-b455-4973b6627bd5"). InnerVolumeSpecName "kube-api-access-mqxxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:26.721389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.715436    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqxxw\" (UniqueName: \"kubernetes.io/projected/6b27e232-7a42-4130-b455-4973b6627bd5-kube-api-access-mqxxw\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:26.721389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.718268    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:26.721389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.718426    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:26.721389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.718486    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:26.850439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:26.850415    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266" path="/var/lib/kubelet/pods/ac436667-7100-41f1-b3ca-1bc1029f3266/volumes"
Nov 02 01:30:27.155402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.154769    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj" event={"ID":"209f7830-55fc-48b9-8347-5bd2c24ce5e2","Type":"ContainerStarted","Data":"62529d00135af65a8690c4336ceb0e4a3ff57ff34f372817876611ff3cb7aae1"}
Nov 02 01:30:27.155402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.154840    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj" event={"ID":"209f7830-55fc-48b9-8347-5bd2c24ce5e2","Type":"ContainerStarted","Data":"acbfb4c73b910ff8a4231e67fdb1108475b7fe4273a2e5308efa6d7f15e130a4"}
Nov 02 01:30:27.165639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.165605    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp" event={"ID":"6b27e232-7a42-4130-b455-4973b6627bd5","Type":"ContainerDied","Data":"4fc35466c5460c71db9976cd1192197b76069ddfa4733ef7cdac2da8eb4dd8d1"}
Nov 02 01:30:27.165875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.165859    4851 scope.go:122] "RemoveContainer" containerID="26737f40ff736176664324146bb81e235d3e43b791e31b9e9a9c05fef2a4bf70"
Nov 02 01:30:27.166073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.166060    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"
Nov 02 01:30:27.171364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.171269    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" event={"ID":"83e32e09-9992-4e33-906c-cb088eb20ddc","Type":"ContainerStarted","Data":"7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b"}
Nov 02 01:30:27.171499 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.171403    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" event={"ID":"83e32e09-9992-4e33-906c-cb088eb20ddc","Type":"ContainerStarted","Data":"afff7bfb6a1c65ec470c792b404473b91194b66ae192caa4ddfe315d39ccd3d3"}
Nov 02 01:30:27.178370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.178028    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" event={"ID":"afc5860f-cfc3-4caf-844b-92dcbe2d91e4","Type":"ContainerStarted","Data":"aa118a256d9aaf6d1f450aa0b776ea3efa3c0772fc98dbb423dce4dac99a6bee"}
Nov 02 01:30:27.179750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.179695    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj" podStartSLOduration=3.179681837 podStartE2EDuration="3.179681837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:27.174896792 +0000 UTC m=+932.479829620" watchObservedRunningTime="2026-11-02 01:30:27.179681837 +0000 UTC m=+932.484614664"
Nov 02 01:30:27.204362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.203369    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" podStartSLOduration=2.203349964 podStartE2EDuration="2.203349964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:27.198774208 +0000 UTC m=+932.503707035" watchObservedRunningTime="2026-11-02 01:30:27.203349964 +0000 UTC m=+932.508282790"
Nov 02 01:30:27.227490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.227456    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"]
Nov 02 01:30:27.230464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.230420    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:27.242602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.242529    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893050-8l9pp"]
Nov 02 01:30:27.286180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.286125    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podStartSLOduration=3.286109238 podStartE2EDuration="3.286109238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:27.282898206 +0000 UTC m=+932.587831034" watchObservedRunningTime="2026-11-02 01:30:27.286109238 +0000 UTC m=+932.591042064"
Nov 02 01:30:27.301436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.301121    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:27.305976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.305953    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:30:27.306166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.306143    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:30:27.722336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.722306    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"
Nov 02 01:30:27.726282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.726257    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"
Nov 02 01:30:27.791993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.791958    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-maas-api-service-ca\") pod \"d68e7f7a-71a8-47c1-9160-1b35c83e22a6\" (UID: \"d68e7f7a-71a8-47c1-9160-1b35c83e22a6\") "
Nov 02 01:30:27.792205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.792079    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27fb8c50-9ff8-4be1-b961-6f2474b48804-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27fb8c50-9ff8-4be1-b961-6f2474b48804-maas-api-service-ca\") pod \"27fb8c50-9ff8-4be1-b961-6f2474b48804\" (UID: \"27fb8c50-9ff8-4be1-b961-6f2474b48804\") "
Nov 02 01:30:27.792281 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.792256    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27fb8c50-9ff8-4be1-b961-6f2474b48804-kube-api-access-g6jmk\" (UniqueName: \"kubernetes.io/projected/27fb8c50-9ff8-4be1-b961-6f2474b48804-kube-api-access-g6jmk\") pod \"27fb8c50-9ff8-4be1-b961-6f2474b48804\" (UID: \"27fb8c50-9ff8-4be1-b961-6f2474b48804\") "
Nov 02 01:30:27.792436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.792347    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-kube-api-access-rvfsb\" (UniqueName: \"kubernetes.io/projected/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-kube-api-access-rvfsb\") pod \"d68e7f7a-71a8-47c1-9160-1b35c83e22a6\" (UID: \"d68e7f7a-71a8-47c1-9160-1b35c83e22a6\") "
Nov 02 01:30:27.793430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.793407    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fb8c50-9ff8-4be1-b961-6f2474b48804-maas-api-service-ca" pod "27fb8c50-9ff8-4be1-b961-6f2474b48804" (UID: "27fb8c50-9ff8-4be1-b961-6f2474b48804"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:27.793937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.793910    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-maas-api-service-ca" pod "d68e7f7a-71a8-47c1-9160-1b35c83e22a6" (UID: "d68e7f7a-71a8-47c1-9160-1b35c83e22a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:27.795772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.795742    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-kube-api-access-rvfsb" pod "d68e7f7a-71a8-47c1-9160-1b35c83e22a6" (UID: "d68e7f7a-71a8-47c1-9160-1b35c83e22a6"). InnerVolumeSpecName "kube-api-access-rvfsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:27.795947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.795933    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fb8c50-9ff8-4be1-b961-6f2474b48804-kube-api-access-g6jmk" pod "27fb8c50-9ff8-4be1-b961-6f2474b48804" (UID: "27fb8c50-9ff8-4be1-b961-6f2474b48804"). InnerVolumeSpecName "kube-api-access-g6jmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:27.894662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.894581    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6jmk\" (UniqueName: \"kubernetes.io/projected/27fb8c50-9ff8-4be1-b961-6f2474b48804-kube-api-access-g6jmk\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:27.894662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.894610    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfsb\" (UniqueName: \"kubernetes.io/projected/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-kube-api-access-rvfsb\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:27.894662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.894620    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68e7f7a-71a8-47c1-9160-1b35c83e22a6-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:27.894662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:27.894630    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27fb8c50-9ff8-4be1-b961-6f2474b48804-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:28.040317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.038255    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:28.052517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.050585    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:28.151415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.144343    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:28.302918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.302740    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:30:28.303809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.303034    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:30:28.327171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.326765    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2" event={"ID":"d68e7f7a-71a8-47c1-9160-1b35c83e22a6","Type":"ContainerDied","Data":"3ac4d6305662d234909f52f6a75bdee03563f02171b697aa3ad2fda4b0d5c940"}
Nov 02 01:30:28.327171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.326851    4851 scope.go:122] "RemoveContainer" containerID="cfaa53cf8b82a1213fe821068476a311f2e7bcb8ddf52a0d72a4d711095d175a"
Nov 02 01:30:28.327466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.326736    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"
Nov 02 01:30:28.329829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.329806    4851 generic.go:356] "Generic (PLEG): container finished" podID="209f7830-55fc-48b9-8347-5bd2c24ce5e2" containerID="62529d00135af65a8690c4336ceb0e4a3ff57ff34f372817876611ff3cb7aae1" exitCode=0
Nov 02 01:30:28.329930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.329889    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj" event={"ID":"209f7830-55fc-48b9-8347-5bd2c24ce5e2","Type":"ContainerDied","Data":"62529d00135af65a8690c4336ceb0e4a3ff57ff34f372817876611ff3cb7aae1"}
Nov 02 01:30:28.332117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.331968    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg" event={"ID":"27fb8c50-9ff8-4be1-b961-6f2474b48804","Type":"ContainerDied","Data":"1c70b66a5fb21513c3dafb3929cd89aceb9616fd369970f6b03eb9ae8d23d891"}
Nov 02 01:30:28.332117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.332051    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"
Nov 02 01:30:28.343604 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.342122    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:28.345278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.345196    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:30:28.374618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.374420    4851 scope.go:122] "RemoveContainer" containerID="2ec83072daeed7ffcfa77af89b7dec5a4d0bd89d4c1f6d2519c1f9c9e5f888b1"
Nov 02 01:30:28.421552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.421507    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"]
Nov 02 01:30:28.421943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.421909    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:30:28.426880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.425931    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:30:28.429403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.429319    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:30:28.431072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.431038    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893050-7zst2"]
Nov 02 01:30:28.473244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.473168    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"]
Nov 02 01:30:28.480351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.480326    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893050-59qsg"]
Nov 02 01:30:28.867092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.867009    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804" path="/var/lib/kubelet/pods/27fb8c50-9ff8-4be1-b961-6f2474b48804/volumes"
Nov 02 01:30:28.867552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.867509    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b27e232-7a42-4130-b455-4973b6627bd5" path="/var/lib/kubelet/pods/6b27e232-7a42-4130-b455-4973b6627bd5/volumes"
Nov 02 01:30:28.867913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:28.867898    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" path="/var/lib/kubelet/pods/d68e7f7a-71a8-47c1-9160-1b35c83e22a6/volumes"
Nov 02 01:30:29.121909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.121831    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:29.301011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.300977    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:30:29.301199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.301043    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:30:29.379374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.379303    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:29.619384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.614793    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:29.766026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.765978    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:29.777078 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.777059    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:29.881935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.881905    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-xks-ca\") pod \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") "
Nov 02 01:30:29.882459 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.881957    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209f7830-55fc-48b9-8347-5bd2c24ce5e2-kube-api-access-sfvxk\" (UniqueName: \"kubernetes.io/projected/209f7830-55fc-48b9-8347-5bd2c24ce5e2-kube-api-access-sfvxk\") pod \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") "
Nov 02 01:30:29.882459 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.882164    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-openshift-ca\") pod \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\" (UID: \"209f7830-55fc-48b9-8347-5bd2c24ce5e2\") "
Nov 02 01:30:29.882569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.882476    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-openshift-ca" pod "209f7830-55fc-48b9-8347-5bd2c24ce5e2" (UID: "209f7830-55fc-48b9-8347-5bd2c24ce5e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:29.882887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.882865    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:29.884457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.884427    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209f7830-55fc-48b9-8347-5bd2c24ce5e2-kube-api-access-sfvxk" pod "209f7830-55fc-48b9-8347-5bd2c24ce5e2" (UID: "209f7830-55fc-48b9-8347-5bd2c24ce5e2"). InnerVolumeSpecName "kube-api-access-sfvxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:29.884629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.884613    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-xks-ca" pod "209f7830-55fc-48b9-8347-5bd2c24ce5e2" (UID: "209f7830-55fc-48b9-8347-5bd2c24ce5e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:30:29.999701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.999574    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/209f7830-55fc-48b9-8347-5bd2c24ce5e2-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:29.999701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:29.999605    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfvxk\" (UniqueName: \"kubernetes.io/projected/209f7830-55fc-48b9-8347-5bd2c24ce5e2-kube-api-access-sfvxk\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:30.301171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:30.301091    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:30:30.301171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:30.301141    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:30:30.376727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:30.376690    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"
Nov 02 01:30:30.376727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:30.376715    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj" event={"ID":"209f7830-55fc-48b9-8347-5bd2c24ce5e2","Type":"ContainerDied","Data":"acbfb4c73b910ff8a4231e67fdb1108475b7fe4273a2e5308efa6d7f15e130a4"}
Nov 02 01:30:30.376942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:30.376767    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acbfb4c73b910ff8a4231e67fdb1108475b7fe4273a2e5308efa6d7f15e130a4"
Nov 02 01:30:30.793936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:30.793898    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:30.994060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:30.994031    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:31.028195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.028154    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"]
Nov 02 01:30:31.028891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.028873    4851 cpu_manager.go:414] "containerMap: removing container" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerName="cleanup"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.028897    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerName="cleanup"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.028924    4851 cpu_manager.go:414] "containerMap: removing container" podUID="6b27e232-7a42-4130-b455-4973b6627bd5" containerName="cleanup"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.028932    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b27e232-7a42-4130-b455-4973b6627bd5" containerName="cleanup"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.028945    4851 cpu_manager.go:414] "containerMap: removing container" podUID="6b27e232-7a42-4130-b455-4973b6627bd5" containerName="cleanup"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.028953    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b27e232-7a42-4130-b455-4973b6627bd5" containerName="cleanup"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.028979    4851 cpu_manager.go:414] "containerMap: removing container" podUID="209f7830-55fc-48b9-8347-5bd2c24ce5e2" containerName="revoke-keys"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.028986    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209f7830-55fc-48b9-8347-5bd2c24ce5e2" containerName="revoke-keys"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029004    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerName="cleanup"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029011    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerName="cleanup"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029026    4851 cpu_manager.go:414] "containerMap: removing container" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerName="cleanup"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029032    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerName="cleanup"
Nov 02 01:30:31.029037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029042    4851 cpu_manager.go:414] "containerMap: removing container" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029049    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029080    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029087    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029118    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029125    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029279    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029309    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b27e232-7a42-4130-b455-4973b6627bd5" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029318    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b27e232-7a42-4130-b455-4973b6627bd5" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029329    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="209f7830-55fc-48b9-8347-5bd2c24ce5e2" containerName="revoke-keys"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029335    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029341    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac436667-7100-41f1-b3ca-1bc1029f3266" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029348    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029354    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b27e232-7a42-4130-b455-4973b6627bd5" containerName="cleanup"
Nov 02 01:30:31.029784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.029361    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerName="cleanup"
Nov 02 01:30:31.056364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.044642    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:30:31.070375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.069237    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-616e3ed7\""
Nov 02 01:30:31.070375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.069419    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-616e3ed7-dockercfg-j2bv9\""
Nov 02 01:30:31.072658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.072629    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"]
Nov 02 01:30:31.093464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.093436    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f7c8f-8941-4921-8103-82a2414b77f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4\" (UID: \"cd8f7c8f-8941-4921-8103-82a2414b77f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:30:31.093757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.093736    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mj4b\" (UniqueName: \"kubernetes.io/projected/cd8f7c8f-8941-4921-8103-82a2414b77f0-kube-api-access-9mj4b\") pod \"payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4\" (UID: \"cd8f7c8f-8941-4921-8103-82a2414b77f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:30:31.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.123329    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:31.215426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.215386    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f7c8f-8941-4921-8103-82a2414b77f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4\" (UID: \"cd8f7c8f-8941-4921-8103-82a2414b77f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:30:31.215908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.215883    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mj4b\" (UniqueName: \"kubernetes.io/projected/cd8f7c8f-8941-4921-8103-82a2414b77f0-kube-api-access-9mj4b\") pod \"payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4\" (UID: \"cd8f7c8f-8941-4921-8103-82a2414b77f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:30:31.239317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.217423    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f7c8f-8941-4921-8103-82a2414b77f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4\" (UID: \"cd8f7c8f-8941-4921-8103-82a2414b77f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:30:31.302319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.298244    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mj4b\" (UniqueName: \"kubernetes.io/projected/cd8f7c8f-8941-4921-8103-82a2414b77f0-kube-api-access-9mj4b\") pod \"payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4\" (UID: \"cd8f7c8f-8941-4921-8103-82a2414b77f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:30:31.302319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.300740    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:30:31.302319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.300790    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:30:31.409370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.390581    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:30:31.864017 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:31.863983    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8f7c8f_8941_4921_8103_82a2414b77f0.slice/crio-4ffabdd30096b5708adf64630547dc778e7233406512fabfb947239081d5b7f3 WatchSource:0}: Error finding container 4ffabdd30096b5708adf64630547dc778e7233406512fabfb947239081d5b7f3: Status 404 returned error can't find the container with id 4ffabdd30096b5708adf64630547dc778e7233406512fabfb947239081d5b7f3
Nov 02 01:30:31.866971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:31.866942    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"]
Nov 02 01:30:32.075766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:32.075730    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:32.301143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:32.301110    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:30:32.301368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:32.301163    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:30:32.577367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:32.577029    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4" event={"ID":"cd8f7c8f-8941-4921-8103-82a2414b77f0","Type":"ContainerStarted","Data":"3f17d507c20f5ea6eb52ffadad28ee45c03ba8143f9cddf3dc3d3ea052616425"}
Nov 02 01:30:32.577367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:32.577092    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4" event={"ID":"cd8f7c8f-8941-4921-8103-82a2414b77f0","Type":"ContainerStarted","Data":"4ffabdd30096b5708adf64630547dc778e7233406512fabfb947239081d5b7f3"}
Nov 02 01:30:32.577367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:32.577182    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:30:32.609628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:32.608932    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4" podStartSLOduration=1.608918995 podStartE2EDuration="1.608918995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:32.602639623 +0000 UTC m=+937.907572469" watchObservedRunningTime="2026-11-02 01:30:32.608918995 +0000 UTC m=+937.913851822"
Nov 02 01:30:33.018266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:33.018228    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:33.168174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:33.168134    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:33.300831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:33.300738    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:30:33.300831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:33.300814    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:30:33.334970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:33.334916    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:34.301317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:34.301183    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:30:34.301317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:34.301249    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:30:34.871316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:34.870774    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:35.161411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:35.146653    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:35.238484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:35.238434    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:35.311723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:35.311688    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 01:30:35.312277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:35.312249    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 01:30:35.586367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:35.583389    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:35.631317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:35.630994    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:35.675441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:35.675391    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:35.822324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:35.778813    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:35.823046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:35.822607    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:36.044379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.044332    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:30:36.074328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.063317    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:30:36.077492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.077457    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:36.078029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.078003    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:36.078229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.078210    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:30:36.207317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.182214    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:36.422528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.422499    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:30:36.609328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.599791    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:36.622419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.622378    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:30:36.720373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.682711    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:36.720373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.683173    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:30:36.720373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.685131    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:30:36.720373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.685384    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:30:36.750522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.749678    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:30:36.752607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.752571    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Nov 02 01:30:36.865318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.843613    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:30:36.892320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.867279    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:30:36.928318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.914706    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:30:36.942794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:36.942756    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:30:37.070475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.070367    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:37.072473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.072438    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:30:37.190227 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.073542    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:37.195328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.103489    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:37.198429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.112413    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:30:37.199726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.112444    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:30:37.430321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.426745    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:37.705375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.691599    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:30:37.705375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.691865    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:30:37.705375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.691932    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:30:37.705375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.691990    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:30:37.705375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.692221    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:30:37.705375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.692890    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:37.705375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.694159    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:30:37.840319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.551003    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:37.840319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.839981    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:37.870319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.869635    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:37.918319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.915892    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:37.928323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:37.926979    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:38.089477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:38.089366    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:30:38.247589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:38.247539    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:38.249009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:38.248957    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:38.267503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:38.267456    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:30:38.382709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:38.380407    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:38.411772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:38.399482    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:38.504319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:38.482093    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:38.749592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:38.749557    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:39.099532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.099433    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:30:39.145641 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.101997    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:30:39.145886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.102051    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:30:39.145989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.102420    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:39.146086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.135944    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:30:39.172318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.147597    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:30:39.236805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.236762    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:39.237109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.237058    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:39.282325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.265855    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:39.559332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.351698    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:30:39.559332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.351741    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:30:39.559332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.387454    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:39.559332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.388809    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:39.559332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.402205    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:39.767065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:39.767023    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded"
Nov 02 01:30:40.329318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.320424    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:40.331834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.331791    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:30:40.353340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.346900    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:30:40.353340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.347932    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:40.353340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.347978    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:40.355231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.355201    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:40.380408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.380363    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:40.394385 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.391468    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:30:40.394949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.394910    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:40.659373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.589562    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:40.688729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.657892    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:40.717054 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.657927    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:30:40.718443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.657948    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:40.741545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.657967    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:30:40.741545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.657995    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:30:40.741545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.658380    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:40.801704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:40.801629    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:41.123038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.122988    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:30:41.123431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.123390    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:41.185536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.185501    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:41.322706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.322645    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:41.332428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.332385    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:41.357645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.357601    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:41.382000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.381903    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:41.400436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.400385    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:41.401892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.401858    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:30:41.690811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.690768    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:30:41.747481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.692448    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:30:41.747481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.728382    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:41.747481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.732682    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:41.757829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.757788    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:41.758417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.758072    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:41.776995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:41.775656    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:30:42.323429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.323380    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:42.337165 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.337123    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:42.358021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.357979    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:42.563328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.537693    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:42.563328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.540543    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:42.563328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.540949    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:30:42.720360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.704065    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:30:42.721023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.720973    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:30:42.737557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.737491    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:42.746583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.740114    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:43.406364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.746864    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:30:43.406638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.765407    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:43.408128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.765699    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:43.409457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.788659    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:43.409655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:42.829478    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded"
Nov 02 01:30:43.409807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.273535    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:43.409963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.351081    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:30:43.411150 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.351159    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:43.428388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.360510    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:43.559439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.559383    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:30:43.559858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.559828    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:30:43.699997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.680967    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:43.791316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.755131    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:43.813590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.792721    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:43.822826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.811150    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:30:43.833946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:43.833596    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:44.104943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.104811    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:44.127344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.121810    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:44.277011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.276970    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:44.408369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.407965    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:30:44.544317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.430493    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:30:44.544317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.430881    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:30:44.544317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.432091    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:44.544317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.433420    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:44.544317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.446033    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:44.544317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.467689    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:30:44.569161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.568128    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:30:44.701883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.701666    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:44.834317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.829474    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:30:44.834317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.830567    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:44.864862 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:44.864812    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:45.178597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.162047    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl" podUID="437397c1-b8b9-4a7a-90db-17411343e18e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:45.196186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.196038    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9" podUID="d46b2d7a-fabe-4466-818c-9fc511dbd10e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.435691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.417038    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:45.749441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.543437    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.749441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.543475    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.749441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.543496    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.749441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.543532    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.749441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.543551    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.749441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.605368    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr" podUID="d52b9b8d-07d7-49d9-b7e0-92cd937b26c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:45.749441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.619087    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.749441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.685015    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.749441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.706600    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.805144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.805091    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.830162 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.830119    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:45.839428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.839375    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:30:45.890677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.890543    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:30:45.938946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.937754    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:46.009191 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:45.967690    4851 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.091s"
Nov 02 01:30:46.009191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:45.999500    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"]
Nov 02 01:30:46.009191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.000553    4851 cpu_manager.go:414] "containerMap: removing container" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerName="cleanup"
Nov 02 01:30:46.009191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.000569    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerName="cleanup"
Nov 02 01:30:46.009191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.000641    4851 cpu_manager.go:414] "containerMap: removing container" podUID="6b27e232-7a42-4130-b455-4973b6627bd5" containerName="cleanup"
Nov 02 01:30:46.009191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.000649    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b27e232-7a42-4130-b455-4973b6627bd5" containerName="cleanup"
Nov 02 01:30:46.009191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.000826    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68e7f7a-71a8-47c1-9160-1b35c83e22a6" containerName="cleanup"
Nov 02 01:30:46.009191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.000848    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fb8c50-9ff8-4be1-b961-6f2474b48804" containerName="cleanup"
Nov 02 01:30:46.064596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.058224    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:30:46.097357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.077869    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:46.106540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.106224    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:30:46.119068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.119042    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:30:46.119345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.119327    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:30:46.119536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.119522    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:30:46.119729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.119708    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:30:46.126310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.124642    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:30:46.184217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.184181    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-854ec26e-pgkth\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:46.184658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.184638    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-854ec26e-pgkth\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:46.185037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.185022    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bkdn\" (UniqueName: \"kubernetes.io/projected/d99c1761-ee97-4754-97c8-8cf6e7d4236c-kube-api-access-5bkdn\") pod \"maas-api-revoke-keys-e2e-ait-854ec26e-pgkth\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:46.221322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.219377    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"]
Nov 02 01:30:46.221322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.219485    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:30:46.221322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.219518    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:30:46.252378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.252350    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:30:46.257518 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.257495    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:30:46.308537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.308509    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:30:46.318199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.318172    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:30:46.343371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.320939    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:30:46.343371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.320984    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:30:46.343371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.319358    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-854ec26e-pgkth\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:46.343371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.321819    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-854ec26e-pgkth\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:46.343371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.322006    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bkdn\" (UniqueName: \"kubernetes.io/projected/d99c1761-ee97-4754-97c8-8cf6e7d4236c-kube-api-access-5bkdn\") pod \"maas-api-revoke-keys-e2e-ait-854ec26e-pgkth\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:46.343371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.320140    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-854ec26e-pgkth\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:46.343371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.311977    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:46.343371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.326755    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:30:46.423568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.423536    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-854ec26e-pgkth\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:46.426117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.426086    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bkdn\" (UniqueName: \"kubernetes.io/projected/d99c1761-ee97-4754-97c8-8cf6e7d4236c-kube-api-access-5bkdn\") pod \"maas-api-revoke-keys-e2e-ait-854ec26e-pgkth\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:46.525331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.523344    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:46.664450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:46.663856    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:47.071348 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:47.071314    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99c1761_ee97_4754_97c8_8cf6e7d4236c.slice/crio-2516dd34c97f50b2d42f929cac8e64194b79662a5c69a060275d14657ce116db WatchSource:0}: Error finding container 2516dd34c97f50b2d42f929cac8e64194b79662a5c69a060275d14657ce116db: Status 404 returned error can't find the container with id 2516dd34c97f50b2d42f929cac8e64194b79662a5c69a060275d14657ce116db
Nov 02 01:30:47.071803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.071726    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"]
Nov 02 01:30:47.231495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.231458    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:47.581983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.581946    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth" event={"ID":"d99c1761-ee97-4754-97c8-8cf6e7d4236c","Type":"ContainerStarted","Data":"36888737e7a47a361d80260372b8a32bdf175e7e8acb5843198c7910302780d5"}
Nov 02 01:30:47.584484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.584451    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"]
Nov 02 01:30:47.590845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.590803    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth" event={"ID":"d99c1761-ee97-4754-97c8-8cf6e7d4236c","Type":"ContainerStarted","Data":"2516dd34c97f50b2d42f929cac8e64194b79662a5c69a060275d14657ce116db"}
Nov 02 01:30:47.592847 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.591122    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.617684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.613448    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwn4\" (UniqueName: \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-kube-api-access-hrwn4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.617684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.615050    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.617684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.615088    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad85307-4fc4-4708-8153-00e8a0279d76-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.617684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.615413    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.617684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.615492    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.617684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.615542    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.617684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.615662    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.626452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.623429    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"]
Nov 02 01:30:47.626739 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.626691    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth" podStartSLOduration=2.62666455 podStartE2EDuration="2.62666455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:47.618676308 +0000 UTC m=+952.923609134" watchObservedRunningTime="2026-11-02 01:30:47.62666455 +0000 UTC m=+952.931597376"
Nov 02 01:30:47.674432 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:30:47.674389    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:30:47.717635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.717546    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.717635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.717596    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad85307-4fc4-4708-8153-00e8a0279d76-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.717878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.717766    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.717878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.717801    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.717878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.717840    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.718037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.717916    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.718091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.718065    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwn4\" (UniqueName: \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-kube-api-access-hrwn4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.719111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.719080    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.721252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.721226    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad85307-4fc4-4708-8153-00e8a0279d76-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.722088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.722059    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.723937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.723911    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.724142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.724121    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.730746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.730712    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwn4\" (UniqueName: \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-kube-api-access-hrwn4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.730864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.730837    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvzjh\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:47.929560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:47.929526    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:48.031880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.031551    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:48.244589 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:48.244428    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad85307_4fc4_4708_8153_00e8a0279d76.slice/crio-dd33a836d4db3f2e993bd53761f6b11df86e70609ad0943ddc1e8b40f4af1681 WatchSource:0}: Error finding container dd33a836d4db3f2e993bd53761f6b11df86e70609ad0943ddc1e8b40f4af1681: Status 404 returned error can't find the container with id dd33a836d4db3f2e993bd53761f6b11df86e70609ad0943ddc1e8b40f4af1681
Nov 02 01:30:48.247706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.247656    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:48.247806 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.247739    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:48.249402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.249378    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"]
Nov 02 01:30:48.645932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.638989    4851 generic.go:356] "Generic (PLEG): container finished" podID="d99c1761-ee97-4754-97c8-8cf6e7d4236c" containerID="36888737e7a47a361d80260372b8a32bdf175e7e8acb5843198c7910302780d5" exitCode=0
Nov 02 01:30:48.645932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.639213    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth" event={"ID":"d99c1761-ee97-4754-97c8-8cf6e7d4236c","Type":"ContainerDied","Data":"36888737e7a47a361d80260372b8a32bdf175e7e8acb5843198c7910302780d5"}
Nov 02 01:30:48.645932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.644079    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh" event={"ID":"dad85307-4fc4-4708-8153-00e8a0279d76","Type":"ContainerStarted","Data":"996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0"}
Nov 02 01:30:48.645932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.644171    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh" event={"ID":"dad85307-4fc4-4708-8153-00e8a0279d76","Type":"ContainerStarted","Data":"dd33a836d4db3f2e993bd53761f6b11df86e70609ad0943ddc1e8b40f4af1681"}
Nov 02 01:30:48.647419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.647385    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:48.723080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.723006    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh" podStartSLOduration=1.72298522 podStartE2EDuration="1.72298522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:48.722487225 +0000 UTC m=+954.027420060" watchObservedRunningTime="2026-11-02 01:30:48.72298522 +0000 UTC m=+954.027918053"
Nov 02 01:30:48.934324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:48.933456    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:49.083703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:49.083161    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:49.684473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:49.684435    4851 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mvzjh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:30:49.684875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:49.684502    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh" podUID="dad85307-4fc4-4708-8153-00e8a0279d76" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:30:49.762209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:49.762177    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:49.992897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:49.992874    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:50.030613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.030577    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-xks-ca\") pod \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") "
Nov 02 01:30:50.030812 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.030665    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-openshift-ca\") pod \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") "
Nov 02 01:30:50.031062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.031042    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99c1761-ee97-4754-97c8-8cf6e7d4236c-kube-api-access-5bkdn\" (UniqueName: \"kubernetes.io/projected/d99c1761-ee97-4754-97c8-8cf6e7d4236c-kube-api-access-5bkdn\") pod \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\" (UID: \"d99c1761-ee97-4754-97c8-8cf6e7d4236c\") "
Nov 02 01:30:50.031173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.031091    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-openshift-ca" pod "d99c1761-ee97-4754-97c8-8cf6e7d4236c" (UID: "d99c1761-ee97-4754-97c8-8cf6e7d4236c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:50.031832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.031805    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.034110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.034081    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99c1761-ee97-4754-97c8-8cf6e7d4236c-kube-api-access-5bkdn" pod "d99c1761-ee97-4754-97c8-8cf6e7d4236c" (UID: "d99c1761-ee97-4754-97c8-8cf6e7d4236c"). InnerVolumeSpecName "kube-api-access-5bkdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:50.034651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.034621    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-xks-ca" pod "d99c1761-ee97-4754-97c8-8cf6e7d4236c" (UID: "d99c1761-ee97-4754-97c8-8cf6e7d4236c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:30:50.133201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.133163    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bkdn\" (UniqueName: \"kubernetes.io/projected/d99c1761-ee97-4754-97c8-8cf6e7d4236c-kube-api-access-5bkdn\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.133201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.133197    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d99c1761-ee97-4754-97c8-8cf6e7d4236c-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.555328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.552604    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:50.698355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.698320    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"
Nov 02 01:30:50.698992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.698964    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth" event={"ID":"d99c1761-ee97-4754-97c8-8cf6e7d4236c","Type":"ContainerDied","Data":"2516dd34c97f50b2d42f929cac8e64194b79662a5c69a060275d14657ce116db"}
Nov 02 01:30:50.699088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.699022    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2516dd34c97f50b2d42f929cac8e64194b79662a5c69a060275d14657ce116db"
Nov 02 01:30:50.699625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.699603    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:30:50.745313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.744534    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"]
Nov 02 01:30:50.751350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.745693    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d99c1761-ee97-4754-97c8-8cf6e7d4236c" containerName="revoke-keys"
Nov 02 01:30:50.751350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.745714    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d99c1761-ee97-4754-97c8-8cf6e7d4236c" containerName="revoke-keys"
Nov 02 01:30:50.751350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.745859    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d99c1761-ee97-4754-97c8-8cf6e7d4236c" containerName="revoke-keys"
Nov 02 01:30:50.752707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.752687    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:50.761461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.756084    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"]
Nov 02 01:30:50.761461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.757334    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:30:50.761461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.757438    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:30:50.761461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.757492    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:30:50.761461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.758021    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:30:50.863968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.863859    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvh6\" (UniqueName: \"kubernetes.io/projected/26f51a4f-df18-45c9-ad0f-de40ba621ef7-kube-api-access-mjvh6\") pod \"maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:50.881439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.864279    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:50.881619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.881518    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:50.983065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.983013    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:50.983272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.983088    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:50.983407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.983357    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvh6\" (UniqueName: \"kubernetes.io/projected/26f51a4f-df18-45c9-ad0f-de40ba621ef7-kube-api-access-mjvh6\") pod \"maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:50.984259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.984233    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:50.986190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.986165    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:50.996051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:50.996023    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvh6\" (UniqueName: \"kubernetes.io/projected/26f51a4f-df18-45c9-ad0f-de40ba621ef7-kube-api-access-mjvh6\") pod \"maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:51.076139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:51.076101    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:51.465937 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:30:51.465901    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f51a4f_df18_45c9_ad0f_de40ba621ef7.slice/crio-11212d10bb6d5e7a677fce9a917cff8a0218efe8a57ba5845dc43f26b9bf6c8d WatchSource:0}: Error finding container 11212d10bb6d5e7a677fce9a917cff8a0218efe8a57ba5845dc43f26b9bf6c8d: Status 404 returned error can't find the container with id 11212d10bb6d5e7a677fce9a917cff8a0218efe8a57ba5845dc43f26b9bf6c8d
Nov 02 01:30:51.468434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:51.468408    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"]
Nov 02 01:30:51.533395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:51.533334    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:51.710198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:51.710164    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv" event={"ID":"26f51a4f-df18-45c9-ad0f-de40ba621ef7","Type":"ContainerStarted","Data":"11212d10bb6d5e7a677fce9a917cff8a0218efe8a57ba5845dc43f26b9bf6c8d"}
Nov 02 01:30:52.451056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:52.451023    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:52.723049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:52.722951    4851 generic.go:356] "Generic (PLEG): container finished" podID="26f51a4f-df18-45c9-ad0f-de40ba621ef7" containerID="dd6b7ca333c575bc4734fe89fb813194f99b84d8ef0586cdf06e8d46fec2a338" exitCode=0
Nov 02 01:30:52.723049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:52.723008    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv" event={"ID":"26f51a4f-df18-45c9-ad0f-de40ba621ef7","Type":"ContainerDied","Data":"dd6b7ca333c575bc4734fe89fb813194f99b84d8ef0586cdf06e8d46fec2a338"}
Nov 02 01:30:53.557331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:53.556586    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:53.938605 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:53.938578    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:54.058983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.058950    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f51a4f-df18-45c9-ad0f-de40ba621ef7-kube-api-access-mjvh6\" (UniqueName: \"kubernetes.io/projected/26f51a4f-df18-45c9-ad0f-de40ba621ef7-kube-api-access-mjvh6\") pod \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") "
Nov 02 01:30:54.059275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.059259    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-openshift-ca\") pod \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") "
Nov 02 01:30:54.059446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.059428    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-xks-ca\") pod \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\" (UID: \"26f51a4f-df18-45c9-ad0f-de40ba621ef7\") "
Nov 02 01:30:54.059678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.059656    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-openshift-ca" pod "26f51a4f-df18-45c9-ad0f-de40ba621ef7" (UID: "26f51a4f-df18-45c9-ad0f-de40ba621ef7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:30:54.060092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.060037    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:54.061508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.061452    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f51a4f-df18-45c9-ad0f-de40ba621ef7-kube-api-access-mjvh6" pod "26f51a4f-df18-45c9-ad0f-de40ba621ef7" (UID: "26f51a4f-df18-45c9-ad0f-de40ba621ef7"). InnerVolumeSpecName "kube-api-access-mjvh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:54.061647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.061632    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-xks-ca" pod "26f51a4f-df18-45c9-ad0f-de40ba621ef7" (UID: "26f51a4f-df18-45c9-ad0f-de40ba621ef7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:30:54.162361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.162322    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f51a4f-df18-45c9-ad0f-de40ba621ef7-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:54.162361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.162363    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjvh6\" (UniqueName: \"kubernetes.io/projected/26f51a4f-df18-45c9-ad0f-de40ba621ef7-kube-api-access-mjvh6\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:30:54.585041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.585008    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:54.749537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.749503    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv" event={"ID":"26f51a4f-df18-45c9-ad0f-de40ba621ef7","Type":"ContainerDied","Data":"11212d10bb6d5e7a677fce9a917cff8a0218efe8a57ba5845dc43f26b9bf6c8d"}
Nov 02 01:30:54.749724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.749559    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11212d10bb6d5e7a677fce9a917cff8a0218efe8a57ba5845dc43f26b9bf6c8d"
Nov 02 01:30:54.749724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:54.749519    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"
Nov 02 01:30:55.383507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:55.383484    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"]
Nov 02 01:30:55.400353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:55.400315    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"]
Nov 02 01:30:55.491769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:55.491733    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:55.622304 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:55.622203    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:55.804316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:55.802617    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:56.492688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:56.492638    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:57.242048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.242014    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:57.349012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.348966    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"]
Nov 02 01:30:57.351566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.351526    4851 cpu_manager.go:414] "containerMap: removing container" podUID="26f51a4f-df18-45c9-ad0f-de40ba621ef7" containerName="revoke-keys"
Nov 02 01:30:57.351799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.351783    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f51a4f-df18-45c9-ad0f-de40ba621ef7" containerName="revoke-keys"
Nov 02 01:30:57.352511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.352490    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f51a4f-df18-45c9-ad0f-de40ba621ef7" containerName="revoke-keys"
Nov 02 01:30:57.366179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.366148    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:57.373143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.371898    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:30:57.373143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.372013    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:30:57.374095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.373576    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:30:57.404050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.386549    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"]
Nov 02 01:30:57.407392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.405103    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:30:57.479137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.479087    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:57.479474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.479450    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llb6b\" (UniqueName: \"kubernetes.io/projected/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-kube-api-access-llb6b\") pod \"maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:57.479591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.479509    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:57.581929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.581828    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:57.582382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.582062    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llb6b\" (UniqueName: \"kubernetes.io/projected/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-kube-api-access-llb6b\") pod \"maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:57.582382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.582109    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:57.582700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.582671    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:57.584830 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.584811    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:57.594621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.594593    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llb6b\" (UniqueName: \"kubernetes.io/projected/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-kube-api-access-llb6b\") pod \"maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:57.744318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:57.744270    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:30:58.021788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:58.021751    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:58.131550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:58.131520    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"]
Nov 02 01:30:58.875162 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:58.875065    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb" event={"ID":"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a","Type":"ContainerStarted","Data":"abd4720485ceb602748c6203942d4c1396a3dcf5120757c48c8826d41b6642e1"}
Nov 02 01:30:58.875162 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:58.875119    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb" event={"ID":"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a","Type":"ContainerStarted","Data":"567d6dd51267f78b26684e7ab5a255f30197e820096b5cd68841caf57cb4e785"}
Nov 02 01:30:58.888339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:58.887776    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:59.628557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:59.628523    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:30:59.886220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:59.886122    4851 generic.go:356] "Generic (PLEG): container finished" podID="8a39f0b0-5d77-492f-9c27-e92e4dc0de9a" containerID="abd4720485ceb602748c6203942d4c1396a3dcf5120757c48c8826d41b6642e1" exitCode=0
Nov 02 01:30:59.886220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:30:59.886203    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb" event={"ID":"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a","Type":"ContainerDied","Data":"abd4720485ceb602748c6203942d4c1396a3dcf5120757c48c8826d41b6642e1"}
Nov 02 01:31:00.390312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.387577    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn" podUID="489366b9-640d-4d92-95b5-ddb11c36f8be" containerName="payload-processing" containerID="cri-o://a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4" gracePeriod=25
Nov 02 01:31:00.401418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.401380    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2" podUID="a4454f04-826e-45f4-af38-db0bbd64603c" containerName="payload-pre-processing" containerID="cri-o://dcce99b3f1f3801d9f929d81c3c456bf14ade44f3d70df4be10501ec19fcf365" gracePeriod=25
Nov 02 01:31:00.430315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.426702    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:00.881356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.881329    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"]
Nov 02 01:31:00.900652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.900621    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:31:00.916247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.916203    4851 generic.go:356] "Generic (PLEG): container finished" podID="489366b9-640d-4d92-95b5-ddb11c36f8be" containerID="a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4" exitCode=0
Nov 02 01:31:00.916606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.916320    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn" event={"ID":"489366b9-640d-4d92-95b5-ddb11c36f8be","Type":"ContainerDied","Data":"a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4"}
Nov 02 01:31:00.916606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.916377    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn" event={"ID":"489366b9-640d-4d92-95b5-ddb11c36f8be","Type":"ContainerDied","Data":"da099accfb07f89625fb279f781a271ce7b1178f204d82b0e0fd6b4121c86b62"}
Nov 02 01:31:00.916606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.916411    4851 scope.go:122] "RemoveContainer" containerID="a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4"
Nov 02 01:31:00.916801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.916634    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"
Nov 02 01:31:00.922490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.922460    4851 generic.go:356] "Generic (PLEG): container finished" podID="a4454f04-826e-45f4-af38-db0bbd64603c" containerID="dcce99b3f1f3801d9f929d81c3c456bf14ade44f3d70df4be10501ec19fcf365" exitCode=0
Nov 02 01:31:00.922809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.922789    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2" event={"ID":"a4454f04-826e-45f4-af38-db0bbd64603c","Type":"ContainerDied","Data":"dcce99b3f1f3801d9f929d81c3c456bf14ade44f3d70df4be10501ec19fcf365"}
Nov 02 01:31:00.950503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.950350    4851 scope.go:122] "RemoveContainer" containerID="a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4"
Nov 02 01:31:00.965368 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:31:00.951345    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4\": container with ID starting with a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4 not found: ID does not exist" containerID="a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4"
Nov 02 01:31:00.965368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.951492    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4"} err="failed to get container status \"a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4\": rpc error: code = NotFound desc = could not find container \"a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4\": container with ID starting with a018d8f6b5140e2811fe5c8ceea4fa978ef87c712ef1e999cbf877eb049927e4 not found: ID does not exist"
Nov 02 01:31:00.965368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.954629    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:31:00.975266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:00.975234    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:01.000274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.000190    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489366b9-640d-4d92-95b5-ddb11c36f8be-kube-api-access-tm6fx\" (UniqueName: \"kubernetes.io/projected/489366b9-640d-4d92-95b5-ddb11c36f8be-kube-api-access-tm6fx\") pod \"489366b9-640d-4d92-95b5-ddb11c36f8be\" (UID: \"489366b9-640d-4d92-95b5-ddb11c36f8be\") "
Nov 02 01:31:01.000527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.000510    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489366b9-640d-4d92-95b5-ddb11c36f8be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489366b9-640d-4d92-95b5-ddb11c36f8be-plugins-config-volume\") pod \"489366b9-640d-4d92-95b5-ddb11c36f8be\" (UID: \"489366b9-640d-4d92-95b5-ddb11c36f8be\") "
Nov 02 01:31:01.001275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.001252    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489366b9-640d-4d92-95b5-ddb11c36f8be-plugins-config-volume" pod "489366b9-640d-4d92-95b5-ddb11c36f8be" (UID: "489366b9-640d-4d92-95b5-ddb11c36f8be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:31:01.004199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.004165    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489366b9-640d-4d92-95b5-ddb11c36f8be-kube-api-access-tm6fx" pod "489366b9-640d-4d92-95b5-ddb11c36f8be" (UID: "489366b9-640d-4d92-95b5-ddb11c36f8be"). InnerVolumeSpecName "kube-api-access-tm6fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:01.102252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.102214    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4454f04-826e-45f4-af38-db0bbd64603c-kube-api-access-gstqc\" (UniqueName: \"kubernetes.io/projected/a4454f04-826e-45f4-af38-db0bbd64603c-kube-api-access-gstqc\") pod \"a4454f04-826e-45f4-af38-db0bbd64603c\" (UID: \"a4454f04-826e-45f4-af38-db0bbd64603c\") "
Nov 02 01:31:01.102474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.102375    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4454f04-826e-45f4-af38-db0bbd64603c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4454f04-826e-45f4-af38-db0bbd64603c-plugins-config-volume\") pod \"a4454f04-826e-45f4-af38-db0bbd64603c\" (UID: \"a4454f04-826e-45f4-af38-db0bbd64603c\") "
Nov 02 01:31:01.102815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.102790    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4454f04-826e-45f4-af38-db0bbd64603c-plugins-config-volume" pod "a4454f04-826e-45f4-af38-db0bbd64603c" (UID: "a4454f04-826e-45f4-af38-db0bbd64603c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:31:01.103809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.103588    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489366b9-640d-4d92-95b5-ddb11c36f8be-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:01.103809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.103618    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4454f04-826e-45f4-af38-db0bbd64603c-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:01.103809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.103633    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm6fx\" (UniqueName: \"kubernetes.io/projected/489366b9-640d-4d92-95b5-ddb11c36f8be-kube-api-access-tm6fx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:01.104666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.104637    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:01.106087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.106057    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4454f04-826e-45f4-af38-db0bbd64603c-kube-api-access-gstqc" pod "a4454f04-826e-45f4-af38-db0bbd64603c" (UID: "a4454f04-826e-45f4-af38-db0bbd64603c"). InnerVolumeSpecName "kube-api-access-gstqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:01.149803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.149779    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:31:01.212449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.210710    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstqc\" (UniqueName: \"kubernetes.io/projected/a4454f04-826e-45f4-af38-db0bbd64603c-kube-api-access-gstqc\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:01.330315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.313455    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-openshift-ca\") pod \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") "
Nov 02 01:31:01.330315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.313589    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-xks-ca\") pod \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") "
Nov 02 01:31:01.330315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.313736    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-kube-api-access-llb6b\" (UniqueName: \"kubernetes.io/projected/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-kube-api-access-llb6b\") pod \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\" (UID: \"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a\") "
Nov 02 01:31:01.330315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.329061    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-openshift-ca" pod "8a39f0b0-5d77-492f-9c27-e92e4dc0de9a" (UID: "8a39f0b0-5d77-492f-9c27-e92e4dc0de9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:31:01.513314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.427988    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:01.513314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.454518    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-kube-api-access-llb6b" pod "8a39f0b0-5d77-492f-9c27-e92e4dc0de9a" (UID: "8a39f0b0-5d77-492f-9c27-e92e4dc0de9a"). InnerVolumeSpecName "kube-api-access-llb6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:01.513314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.461327    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:01.513314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.489446    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-xks-ca" pod "8a39f0b0-5d77-492f-9c27-e92e4dc0de9a" (UID: "8a39f0b0-5d77-492f-9c27-e92e4dc0de9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:31:01.597329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.573352    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llb6b\" (UniqueName: \"kubernetes.io/projected/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-kube-api-access-llb6b\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:01.597329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.573388    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:01.720665 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.720598    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"]
Nov 02 01:31:01.768133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.768099    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-854ec26e-67558d67d7-s4sbn"]
Nov 02 01:31:01.937357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.937272    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"
Nov 02 01:31:01.937357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.937339    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb" event={"ID":"8a39f0b0-5d77-492f-9c27-e92e4dc0de9a","Type":"ContainerDied","Data":"567d6dd51267f78b26684e7ab5a255f30197e820096b5cd68841caf57cb4e785"}
Nov 02 01:31:01.937867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.937403    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="567d6dd51267f78b26684e7ab5a255f30197e820096b5cd68841caf57cb4e785"
Nov 02 01:31:01.940756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.940734    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2" event={"ID":"a4454f04-826e-45f4-af38-db0bbd64603c","Type":"ContainerDied","Data":"eb62bbb27c3adb91489716b553ece01273359d9d0d64a177e5e60d3fe37e961c"}
Nov 02 01:31:01.940891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.940788    4851 scope.go:122] "RemoveContainer" containerID="dcce99b3f1f3801d9f929d81c3c456bf14ade44f3d70df4be10501ec19fcf365"
Nov 02 01:31:01.940891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.940741    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"
Nov 02 01:31:01.975749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.975719    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"]
Nov 02 01:31:01.982061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:01.982039    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-854ec26e-59d6f547f5-bl2t2"]
Nov 02 01:31:02.133926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:02.133890    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:02.273852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:02.273769    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:02.848868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:02.848831    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489366b9-640d-4d92-95b5-ddb11c36f8be" path="/var/lib/kubelet/pods/489366b9-640d-4d92-95b5-ddb11c36f8be/volumes"
Nov 02 01:31:02.849419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:02.849401    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4454f04-826e-45f4-af38-db0bbd64603c" path="/var/lib/kubelet/pods/a4454f04-826e-45f4-af38-db0bbd64603c/volumes"
Nov 02 01:31:02.905000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:02.904968    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:03.636978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:03.636946    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:04.196272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:04.196241    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:04.335373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:04.335343    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:04.385563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:04.385529    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"]
Nov 02 01:31:04.391675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:04.391652    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-854ec26e-pgkth"]
Nov 02 01:31:04.511369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:04.509895    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:04.700339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:04.697568    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:05.241315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:05.237950    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d99c1761-ee97-4754-97c8-8cf6e7d4236c" path="/var/lib/kubelet/pods/d99c1761-ee97-4754-97c8-8cf6e7d4236c/volumes"
Nov 02 01:31:05.289412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:05.287710    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:05.359453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:05.359418    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"]
Nov 02 01:31:05.366907 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:05.366651    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2f21c877-5m2dv"]
Nov 02 01:31:05.419499 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:05.419464    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:05.583251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:05.583142    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:05.750785 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:05.750752    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:05.882909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:05.882816    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf" podUID="e92bea82-bd88-49db-a9cb-c0bb3e51cd42" containerName="payload-processing" containerID="cri-o://55b3ea302f6e8487694c36b97616e0b1c17db9f009e39e0dc58ec08cf22026a3" gracePeriod=25
Nov 02 01:31:06.002166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.002133    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:06.356050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.316861    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:06.542180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.542061    4851 generic.go:356] "Generic (PLEG): container finished" podID="e92bea82-bd88-49db-a9cb-c0bb3e51cd42" containerID="55b3ea302f6e8487694c36b97616e0b1c17db9f009e39e0dc58ec08cf22026a3" exitCode=0
Nov 02 01:31:06.542501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.542463    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf" event={"ID":"e92bea82-bd88-49db-a9cb-c0bb3e51cd42","Type":"ContainerDied","Data":"55b3ea302f6e8487694c36b97616e0b1c17db9f009e39e0dc58ec08cf22026a3"}
Nov 02 01:31:06.563310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.563272    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:31:06.636115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.636035    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-kube-api-access-cwgxw\" (UniqueName: \"kubernetes.io/projected/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-kube-api-access-cwgxw\") pod \"e92bea82-bd88-49db-a9cb-c0bb3e51cd42\" (UID: \"e92bea82-bd88-49db-a9cb-c0bb3e51cd42\") "
Nov 02 01:31:06.636115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.636070    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-plugins-config-volume\") pod \"e92bea82-bd88-49db-a9cb-c0bb3e51cd42\" (UID: \"e92bea82-bd88-49db-a9cb-c0bb3e51cd42\") "
Nov 02 01:31:06.636548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.636526    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-plugins-config-volume" pod "e92bea82-bd88-49db-a9cb-c0bb3e51cd42" (UID: "e92bea82-bd88-49db-a9cb-c0bb3e51cd42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:31:06.636693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.636675    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:06.638274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.638255    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-kube-api-access-cwgxw" pod "e92bea82-bd88-49db-a9cb-c0bb3e51cd42" (UID: "e92bea82-bd88-49db-a9cb-c0bb3e51cd42"). InnerVolumeSpecName "kube-api-access-cwgxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:06.697324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.687087    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:06.740717 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.740676    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwgxw\" (UniqueName: \"kubernetes.io/projected/e92bea82-bd88-49db-a9cb-c0bb3e51cd42-kube-api-access-cwgxw\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:06.831329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.827861    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:06.919337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.916538    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f51a4f-df18-45c9-ad0f-de40ba621ef7" path="/var/lib/kubelet/pods/26f51a4f-df18-45c9-ad0f-de40ba621ef7/volumes"
Nov 02 01:31:06.934421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.928118    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"]
Nov 02 01:31:06.934421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.928389    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj" podUID="50473a1b-e314-4f06-90b3-558dad6dc6cf" containerName="istio-proxy" containerID="cri-o://5ce60ca53d5b22e64363c666dc12edb19941024dab4edd025dfb4f8f95d0361d" gracePeriod=30
Nov 02 01:31:06.981232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:06.981202    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:07.557937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:07.557904    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"
Nov 02 01:31:07.557937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:07.557927    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf" event={"ID":"e92bea82-bd88-49db-a9cb-c0bb3e51cd42","Type":"ContainerDied","Data":"dd5c608340810cc2042abb448724a1c0db06c997f3f4046be376feec80a9b276"}
Nov 02 01:31:07.558216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:07.557996    4851 scope.go:122] "RemoveContainer" containerID="55b3ea302f6e8487694c36b97616e0b1c17db9f009e39e0dc58ec08cf22026a3"
Nov 02 01:31:07.591324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:07.591227    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"]
Nov 02 01:31:07.598576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:07.598536    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:07.604345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:07.604313    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2f21c877-7cc67c68dc-s66rf"]
Nov 02 01:31:07.725763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:07.725732    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:07.883403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:07.883246    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:08.428313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.427970    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"]
Nov 02 01:31:08.447403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447283    4851 cpu_manager.go:414] "containerMap: removing container" podUID="a4454f04-826e-45f4-af38-db0bbd64603c" containerName="payload-pre-processing"
Nov 02 01:31:08.447403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447338    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4454f04-826e-45f4-af38-db0bbd64603c" containerName="payload-pre-processing"
Nov 02 01:31:08.447403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447387    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8a39f0b0-5d77-492f-9c27-e92e4dc0de9a" containerName="revoke-keys"
Nov 02 01:31:08.447403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447397    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a39f0b0-5d77-492f-9c27-e92e4dc0de9a" containerName="revoke-keys"
Nov 02 01:31:08.447403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447445    4851 cpu_manager.go:414] "containerMap: removing container" podUID="e92bea82-bd88-49db-a9cb-c0bb3e51cd42" containerName="payload-processing"
Nov 02 01:31:08.447403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447457    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92bea82-bd88-49db-a9cb-c0bb3e51cd42" containerName="payload-processing"
Nov 02 01:31:08.448163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447492    4851 cpu_manager.go:414] "containerMap: removing container" podUID="489366b9-640d-4d92-95b5-ddb11c36f8be" containerName="payload-processing"
Nov 02 01:31:08.448163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447502    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489366b9-640d-4d92-95b5-ddb11c36f8be" containerName="payload-processing"
Nov 02 01:31:08.448163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447715    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="489366b9-640d-4d92-95b5-ddb11c36f8be" containerName="payload-processing"
Nov 02 01:31:08.448163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447733    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4454f04-826e-45f4-af38-db0bbd64603c" containerName="payload-pre-processing"
Nov 02 01:31:08.448163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447747    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a39f0b0-5d77-492f-9c27-e92e4dc0de9a" containerName="revoke-keys"
Nov 02 01:31:08.448163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.447762    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92bea82-bd88-49db-a9cb-c0bb3e51cd42" containerName="payload-processing"
Nov 02 01:31:08.517371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.504188    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:08.690496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.690328    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.713425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.709860    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:08.715064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.714851    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-89833f31-openshift-default-dockercfg-fpckk\""
Nov 02 01:31:08.798316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.771909    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-token\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.826997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.826960    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-credential-socket\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.827376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.827350    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-envoy\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.827631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.827612    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-socket\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.827795 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.827780    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-data\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.828025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.828008    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbjhd\" (UniqueName: \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-kube-api-access-vbjhd\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.828206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.828190    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-certs\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.828642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.828619    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istiod-ca-cert\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.829138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.829107    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-podinfo\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.932340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.931742    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbjhd\" (UniqueName: \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-kube-api-access-vbjhd\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.932340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.931813    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-certs\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.932340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.931893    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istiod-ca-cert\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.932340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.931967    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-podinfo\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.932340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.932073    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-token\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.932340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.932195    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-credential-socket\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.932340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.932218    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-envoy\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.932340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.932266    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-socket\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.932340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.932343    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-data\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.940322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.933378    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-certs\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.940322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.933991    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istiod-ca-cert\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.940322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.936232    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-data\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.940322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.936559    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-socket\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.940322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.936793    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-credential-socket\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.942645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.942563    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:08.947081 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.946993    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92bea82-bd88-49db-a9cb-c0bb3e51cd42" path="/var/lib/kubelet/pods/e92bea82-bd88-49db-a9cb-c0bb3e51cd42/volumes"
Nov 02 01:31:08.947976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.947931    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"]
Nov 02 01:31:08.990314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.985695    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-token\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:08.990314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:08.986222    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-envoy\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:09.011035 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.010959    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-podinfo\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:09.025387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.025120    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbjhd\" (UniqueName: \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-kube-api-access-vbjhd\") pod \"e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") " pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:09.066364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.065466    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:09.387472 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:31:09.387429    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab8b8cd_bb53_4b88_a82e_6b33f75708a6.slice/crio-dfd9f0421e267cb0a3adc32e9dd4f46e9b046fcdae77989f06404220f4290aca WatchSource:0}: Error finding container dfd9f0421e267cb0a3adc32e9dd4f46e9b046fcdae77989f06404220f4290aca: Status 404 returned error can't find the container with id dfd9f0421e267cb0a3adc32e9dd4f46e9b046fcdae77989f06404220f4290aca
Nov 02 01:31:09.390476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.390440    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:09.390558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.390538    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:09.390616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.390578    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:09.390677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.390614    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"]
Nov 02 01:31:09.717737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.717704    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:09.890357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.889482    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" event={"ID":"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6","Type":"ContainerStarted","Data":"dccd95306e86e67e44813997b9aff23ca1fa1bf1f48a756bb5bf3a033079dc60"}
Nov 02 01:31:09.890357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.889550    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" event={"ID":"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6","Type":"ContainerStarted","Data":"dfd9f0421e267cb0a3adc32e9dd4f46e9b046fcdae77989f06404220f4290aca"}
Nov 02 01:31:09.916637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:09.916571    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" podStartSLOduration=1.9165503350000002 podStartE2EDuration="1.916550335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:09.914278279 +0000 UTC m=+975.219211146" watchObservedRunningTime="2026-11-02 01:31:09.916550335 +0000 UTC m=+975.221483162"
Nov 02 01:31:10.098381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:10.094095    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:10.105995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:10.105962    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:10.236310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:10.236261    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:10.369509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:10.369396    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:10.730594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:10.730554    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:10.864161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:10.864124    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:10.898613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:10.898573    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:10.899713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:10.899691    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:11.024371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:11.023708    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:11.999982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:11.999926    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:12.187543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:12.187506    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:13.232321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:13.225834    4851 generic.go:356] "Generic (PLEG): container finished" podID="50473a1b-e314-4f06-90b3-558dad6dc6cf" containerID="5ce60ca53d5b22e64363c666dc12edb19941024dab4edd025dfb4f8f95d0361d" exitCode=0
Nov 02 01:31:13.262324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:13.259943    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj" event={"ID":"50473a1b-e314-4f06-90b3-558dad6dc6cf","Type":"ContainerDied","Data":"5ce60ca53d5b22e64363c666dc12edb19941024dab4edd025dfb4f8f95d0361d"}
Nov 02 01:31:13.330704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:13.330654    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:13.556440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:13.556352    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:13.835099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:13.835002    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:14.127370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.113799    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:14.163319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.129241    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:31:14.272323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.271849    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:31:14.442425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.376997    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:14.444822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.408683    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:31:14.447052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.409682    4851 patch_prober.go:29] interesting pod/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:31:14.447429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.447400    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj" podUID="50473a1b-e314-4f06-90b3-558dad6dc6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:14.447581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.430553    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:14.448675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.409742    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:31:14.448923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.448903    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:31:14.459274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.453488    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:14.672356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.666841    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:14.717571 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.717461    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:14.738864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.738571    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:31:14.739344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.738630    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:14.769215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:14.769170    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:15.181361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.180367    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:15.238975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.238933    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:15.267861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.267828    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:15.349788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.347058    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-podinfo\") pod \"50473a1b-e314-4f06-90b3-558dad6dc6cf\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") "
Nov 02 01:31:15.349788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.347218    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50473a1b-e314-4f06-90b3-558dad6dc6cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50473a1b-e314-4f06-90b3-558dad6dc6cf-istiod-ca-cert\") pod \"50473a1b-e314-4f06-90b3-558dad6dc6cf\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") "
Nov 02 01:31:15.349788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.347267    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-envoy\") pod \"50473a1b-e314-4f06-90b3-558dad6dc6cf\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") "
Nov 02 01:31:15.361325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.355508    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-socket\") pod \"50473a1b-e314-4f06-90b3-558dad6dc6cf\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") "
Nov 02 01:31:15.361325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.355578    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-credential-socket\") pod \"50473a1b-e314-4f06-90b3-558dad6dc6cf\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") "
Nov 02 01:31:15.361325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.355650    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-certs\") pod \"50473a1b-e314-4f06-90b3-558dad6dc6cf\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") "
Nov 02 01:31:15.361325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.355686    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-kube-api-access-lvtb9\" (UniqueName: \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-kube-api-access-lvtb9\") pod \"50473a1b-e314-4f06-90b3-558dad6dc6cf\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") "
Nov 02 01:31:15.361325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.355773    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-token\" (UniqueName: \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-token\") pod \"50473a1b-e314-4f06-90b3-558dad6dc6cf\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") "
Nov 02 01:31:15.361325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.355816    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-data\") pod \"50473a1b-e314-4f06-90b3-558dad6dc6cf\" (UID: \"50473a1b-e314-4f06-90b3-558dad6dc6cf\") "
Nov 02 01:31:15.361325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.357086    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:31:15.372798 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.372764    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50473a1b-e314-4f06-90b3-558dad6dc6cf-istiod-ca-cert" pod "50473a1b-e314-4f06-90b3-558dad6dc6cf" (UID: "50473a1b-e314-4f06-90b3-558dad6dc6cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:31:15.381129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.381091    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-socket" pod "50473a1b-e314-4f06-90b3-558dad6dc6cf" (UID: "50473a1b-e314-4f06-90b3-558dad6dc6cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:31:15.381563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.381542    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-credential-socket" pod "50473a1b-e314-4f06-90b3-558dad6dc6cf" (UID: "50473a1b-e314-4f06-90b3-558dad6dc6cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:31:15.433770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.410367    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-envoy" pod "50473a1b-e314-4f06-90b3-558dad6dc6cf" (UID: "50473a1b-e314-4f06-90b3-558dad6dc6cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:31:15.433770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.433407    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-data" pod "50473a1b-e314-4f06-90b3-558dad6dc6cf" (UID: "50473a1b-e314-4f06-90b3-558dad6dc6cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:31:15.461318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.460956    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:15.487353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.481531    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:31:15.487353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.481892    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:15.487353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.482611    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:31:15.487353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.482675    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:31:15.525322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.496500    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50473a1b-e314-4f06-90b3-558dad6dc6cf-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:15.525322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.496540    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:15.525322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.496556    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:15.525322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.496572    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:15.525322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.496588    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:15.590544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.590512    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:15.612669 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.608646    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-certs" pod "50473a1b-e314-4f06-90b3-558dad6dc6cf" (UID: "50473a1b-e314-4f06-90b3-558dad6dc6cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:31:15.697378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.659543    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-kube-api-access-lvtb9" pod "50473a1b-e314-4f06-90b3-558dad6dc6cf" (UID: "50473a1b-e314-4f06-90b3-558dad6dc6cf"). InnerVolumeSpecName "kube-api-access-lvtb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:15.697378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.693348    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"]
Nov 02 01:31:15.697378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.693717    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c45ad5e-wd8pb"]
Nov 02 01:31:15.697378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.695004    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50473a1b-e314-4f06-90b3-558dad6dc6cf-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:15.697378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.695071    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvtb9\" (UniqueName: \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-kube-api-access-lvtb9\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:15.698810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.698780    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-podinfo" pod "50473a1b-e314-4f06-90b3-558dad6dc6cf" (UID: "50473a1b-e314-4f06-90b3-558dad6dc6cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:31:15.821882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.790485    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-token" pod "50473a1b-e314-4f06-90b3-558dad6dc6cf" (UID: "50473a1b-e314-4f06-90b3-558dad6dc6cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:15.822103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.759160    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"]
Nov 02 01:31:15.895376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.739806    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:15.895376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.791990    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:15.895376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.844627    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:31:15.895376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.863070    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:15.895376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.863093    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50473a1b-e314-4f06-90b3-558dad6dc6cf-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:15.974369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:15.963513    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg" podUID="9563448c-19ba-4eb3-9ede-9429e5e6a9f2" containerName="istio-proxy" containerID="cri-o://03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee" gracePeriod=30
Nov 02 01:31:16.199504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.199463    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:16.235366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.208907    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:16.365355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.266238    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:16.422753 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.359410    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:16.424360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.367391    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:31:16.424633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.424616    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:31:16.446318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.439708    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:31:16.488901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.463168    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:31:16.488901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.487435    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:16.506351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.490116    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:31:16.506351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.490620    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:16.705317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.701714    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:31:16.751822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.751727    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:31:16.829938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.760242    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:16.855473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.763853    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"
Nov 02 01:31:16.855994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.763871    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj" event={"ID":"50473a1b-e314-4f06-90b3-558dad6dc6cf","Type":"ContainerDied","Data":"1a5bee9fb3c365b9e17f8f53f2ff1cec911adf63630334eeb17ef2ae499e6727"}
Nov 02 01:31:16.870376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.763975    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:16.870839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.856189    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:16.871752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:16.870996    4851 scope.go:122] "RemoveContainer" containerID="5ce60ca53d5b22e64363c666dc12edb19941024dab4edd025dfb4f8f95d0361d"
Nov 02 01:31:17.032374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.029084    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:31:17.041865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.041813    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:17.275316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.267175    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:17.279423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.274154    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a39f0b0-5d77-492f-9c27-e92e4dc0de9a" path="/var/lib/kubelet/pods/8a39f0b0-5d77-492f-9c27-e92e4dc0de9a/volumes"
Nov 02 01:31:17.339495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.339388    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:17.367445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.367398    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:17.433420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.433378    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:31:17.480448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.480388    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:17.509313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.505735    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:17.526819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.526449    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:31:17.527025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.526875    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:31:17.703079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.703039    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:17.704146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.704122    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:31:17.799303 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.798811    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:31:17.830135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.830087    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:31:17.830964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:17.830942    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:31:18.043494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.041486    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:18.043494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.042242    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:31:18.285701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.285659    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"]
Nov 02 01:31:18.343851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.343740    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:18.345433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.345409    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:31:18.352068 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:31:18.345622    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:31:18.488460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.472999    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:18.519310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.507470    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:18.603035 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.602910    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:31:18.603974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.603944    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:31:18.606201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.605928    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:31:18.623865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.623605    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2f21c877-openshift-default-6fc75bdcb9-gz4jj"]
Nov 02 01:31:18.627285 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.622891    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:18.654441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.630191    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:31:18.654441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.631505    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:31:18.654441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.613706    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:18.654441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.632537    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:31:18.694188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.694161    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:31:18.703753 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.703726    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:31:18.726763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.726723    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:31:18.728381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.728363    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:31:18.729655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.729639    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:31:18.806177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.806116    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:31:18.816088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.816052    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:18.919390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:18.915502    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50473a1b-e314-4f06-90b3-558dad6dc6cf" path="/var/lib/kubelet/pods/50473a1b-e314-4f06-90b3-558dad6dc6cf/volumes"
Nov 02 01:31:18.929265 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:31:18.929230    4851 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50473a1b_e314_4f06_90b3_558dad6dc6cf.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50473a1b_e314_4f06_90b3_558dad6dc6cf.slice/crio-1a5bee9fb3c365b9e17f8f53f2ff1cec911adf63630334eeb17ef2ae499e6727\": RecentStats: unable to find data in memory cache]"
Nov 02 01:31:19.093329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:19.087645    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:19.824916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:19.824884    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:19.920360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:19.907347    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"]
Nov 02 01:31:19.920360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:19.908686    4851 cpu_manager.go:414] "containerMap: removing container" podUID="50473a1b-e314-4f06-90b3-558dad6dc6cf" containerName="istio-proxy"
Nov 02 01:31:19.920360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:19.908709    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50473a1b-e314-4f06-90b3-558dad6dc6cf" containerName="istio-proxy"
Nov 02 01:31:19.920360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:19.908939    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="50473a1b-e314-4f06-90b3-558dad6dc6cf" containerName="istio-proxy"
Nov 02 01:31:20.122354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.120435    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"]
Nov 02 01:31:20.122354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.120570    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.129402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.129375    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:31:20.198406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.129754    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:31:20.198638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.129842    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:31:20.198741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.129878    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:31:20.217166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.217127    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af2c41aa-b5mrt\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.217351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.217223    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwfm\" (UniqueName: \"kubernetes.io/projected/471fa604-d12d-44d4-878f-8d6138a47835-kube-api-access-slwfm\") pod \"maas-api-revoke-sub-af2c41aa-b5mrt\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.217472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.217457    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af2c41aa-b5mrt\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.319658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.319624    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af2c41aa-b5mrt\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.319872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.319711    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwfm\" (UniqueName: \"kubernetes.io/projected/471fa604-d12d-44d4-878f-8d6138a47835-kube-api-access-slwfm\") pod \"maas-api-revoke-sub-af2c41aa-b5mrt\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.319937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.319928    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af2c41aa-b5mrt\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.320592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.320570    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af2c41aa-b5mrt\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.329777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.329749    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af2c41aa-b5mrt\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.348326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.344996    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwfm\" (UniqueName: \"kubernetes.io/projected/471fa604-d12d-44d4-878f-8d6138a47835-kube-api-access-slwfm\") pod \"maas-api-revoke-sub-af2c41aa-b5mrt\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.510665 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.510627    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:20.605956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.605864    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:20.777378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:20.767238    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:21.010114 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:21.009075    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:21.496799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:21.489838    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:21.518990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:21.518956    4851 patch_prober.go:29] interesting pod/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:31:21.546968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:21.519237    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg" podUID="9563448c-19ba-4eb3-9ede-9429e5e6a9f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:31:21.650822 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:31:21.549534    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471fa604_d12d_44d4_878f_8d6138a47835.slice/crio-9ea6b2f0f3bbae9f1de0a57c78e5d4bbf8bba24fccf6fe841eb253b2bb3bf3a2 WatchSource:0}: Error finding container 9ea6b2f0f3bbae9f1de0a57c78e5d4bbf8bba24fccf6fe841eb253b2bb3bf3a2: Status 404 returned error can't find the container with id 9ea6b2f0f3bbae9f1de0a57c78e5d4bbf8bba24fccf6fe841eb253b2bb3bf3a2
Nov 02 01:31:21.662266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:21.662232    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"]
Nov 02 01:31:21.898517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:21.898484    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt" event={"ID":"471fa604-d12d-44d4-878f-8d6138a47835","Type":"ContainerStarted","Data":"9ea6b2f0f3bbae9f1de0a57c78e5d4bbf8bba24fccf6fe841eb253b2bb3bf3a2"}
Nov 02 01:31:22.299942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:22.299906    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:22.466383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:22.431863    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:22.584316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:22.580898    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:22.911361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:22.911326    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt" event={"ID":"471fa604-d12d-44d4-878f-8d6138a47835","Type":"ContainerStarted","Data":"58bffe8bdcaf308b8a46d741137e9727eb4a93c39a944febb23ac270cfbd1a1e"}
Nov 02 01:31:22.942943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:22.942888    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt" podStartSLOduration=3.942869018 podStartE2EDuration="3.942869018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:22.929383239 +0000 UTC m=+988.234316066" watchObservedRunningTime="2026-11-02 01:31:22.942869018 +0000 UTC m=+988.247801844"
Nov 02 01:31:23.238831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:23.238795    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:23.990795 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:23.990758    4851 generic.go:356] "Generic (PLEG): container finished" podID="471fa604-d12d-44d4-878f-8d6138a47835" containerID="58bffe8bdcaf308b8a46d741137e9727eb4a93c39a944febb23ac270cfbd1a1e" exitCode=0
Nov 02 01:31:23.991514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:23.991022    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt" event={"ID":"471fa604-d12d-44d4-878f-8d6138a47835","Type":"ContainerDied","Data":"58bffe8bdcaf308b8a46d741137e9727eb4a93c39a944febb23ac270cfbd1a1e"}
Nov 02 01:31:24.044316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:24.040198    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:24.834444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:24.834404    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:25.323649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.323611    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:25.436512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.436479    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-openshift-ca\") pod \"471fa604-d12d-44d4-878f-8d6138a47835\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") "
Nov 02 01:31:25.436512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.436516    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-xks-ca\") pod \"471fa604-d12d-44d4-878f-8d6138a47835\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") "
Nov 02 01:31:25.436775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.436545    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/471fa604-d12d-44d4-878f-8d6138a47835-kube-api-access-slwfm\" (UniqueName: \"kubernetes.io/projected/471fa604-d12d-44d4-878f-8d6138a47835-kube-api-access-slwfm\") pod \"471fa604-d12d-44d4-878f-8d6138a47835\" (UID: \"471fa604-d12d-44d4-878f-8d6138a47835\") "
Nov 02 01:31:25.436955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.436908    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-openshift-ca" pod "471fa604-d12d-44d4-878f-8d6138a47835" (UID: "471fa604-d12d-44d4-878f-8d6138a47835"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:31:25.437504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.437482    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:25.439160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.439136    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-xks-ca" pod "471fa604-d12d-44d4-878f-8d6138a47835" (UID: "471fa604-d12d-44d4-878f-8d6138a47835"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:31:25.439245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.439174    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471fa604-d12d-44d4-878f-8d6138a47835-kube-api-access-slwfm" pod "471fa604-d12d-44d4-878f-8d6138a47835" (UID: "471fa604-d12d-44d4-878f-8d6138a47835"). InnerVolumeSpecName "kube-api-access-slwfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:25.538861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.538827    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/471fa604-d12d-44d4-878f-8d6138a47835-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:25.538861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.538857    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slwfm\" (UniqueName: \"kubernetes.io/projected/471fa604-d12d-44d4-878f-8d6138a47835-kube-api-access-slwfm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:25.629993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:25.629959    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:26.025337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:26.025303    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"
Nov 02 01:31:26.025530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:26.025340    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt" event={"ID":"471fa604-d12d-44d4-878f-8d6138a47835","Type":"ContainerDied","Data":"9ea6b2f0f3bbae9f1de0a57c78e5d4bbf8bba24fccf6fe841eb253b2bb3bf3a2"}
Nov 02 01:31:26.025530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:26.025405    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ea6b2f0f3bbae9f1de0a57c78e5d4bbf8bba24fccf6fe841eb253b2bb3bf3a2"
Nov 02 01:31:26.305199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:26.305121    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:27.034465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:27.034431    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:27.790442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:27.790408    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:28.529446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:28.529406    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:29.254315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:29.252618    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:29.387493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:29.387400    4851 patch_prober.go:29] interesting pod/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:31:29.387493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:29.387461    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:29.993349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:29.993322    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:31:30.069116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.069080    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:30.070803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.070780    4851 generic.go:356] "Generic (PLEG): container finished" podID="9563448c-19ba-4eb3-9ede-9429e5e6a9f2" containerID="03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee" exitCode=0
Nov 02 01:31:30.070972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.070828    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"
Nov 02 01:31:30.070972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.070861    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg" event={"ID":"9563448c-19ba-4eb3-9ede-9429e5e6a9f2","Type":"ContainerDied","Data":"03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee"}
Nov 02 01:31:30.070972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.070917    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg" event={"ID":"9563448c-19ba-4eb3-9ede-9429e5e6a9f2","Type":"ContainerDied","Data":"967913650a6b9e2f9e9b5fd43bee3a8e400d6629c0345498d2829256f48b99cd"}
Nov 02 01:31:30.070972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.070951    4851 scope.go:122] "RemoveContainer" containerID="03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee"
Nov 02 01:31:30.072927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.072905    4851 generic.go:356] "Generic (PLEG): container finished" podID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerID="7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b" exitCode=0
Nov 02 01:31:30.073050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.072987    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" event={"ID":"83e32e09-9992-4e33-906c-cb088eb20ddc","Type":"ContainerDied","Data":"7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b"}
Nov 02 01:31:30.073518 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.073499    4851 scope.go:122] "RemoveContainer" containerID="7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b"
Nov 02 01:31:30.076333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.076258    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:30.076532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.076504    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:30.076673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.076564    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.177513    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istiod-ca-cert\") pod \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") "
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.177636    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-credential-socket\") pod \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") "
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.177685    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-envoy\") pod \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") "
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.177742    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-token\" (UniqueName: \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-token\") pod \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") "
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.177888    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-kube-api-access-jxkx8\" (UniqueName: \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-kube-api-access-jxkx8\") pod \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") "
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.177927    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-certs\") pod \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") "
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.177955    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-socket\") pod \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") "
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.177982    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-data\") pod \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") "
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.178019    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-podinfo\") pod \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\" (UID: \"9563448c-19ba-4eb3-9ede-9429e5e6a9f2\") "
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.180229    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istiod-ca-cert" pod "9563448c-19ba-4eb3-9ede-9429e5e6a9f2" (UID: "9563448c-19ba-4eb3-9ede-9429e5e6a9f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:31:30.185550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.180529    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-credential-socket" pod "9563448c-19ba-4eb3-9ede-9429e5e6a9f2" (UID: "9563448c-19ba-4eb3-9ede-9429e5e6a9f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:31:30.191163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.190277    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-data" pod "9563448c-19ba-4eb3-9ede-9429e5e6a9f2" (UID: "9563448c-19ba-4eb3-9ede-9429e5e6a9f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:31:30.191163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.190528    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-certs" pod "9563448c-19ba-4eb3-9ede-9429e5e6a9f2" (UID: "9563448c-19ba-4eb3-9ede-9429e5e6a9f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:31:30.191163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.190681    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-socket" pod "9563448c-19ba-4eb3-9ede-9429e5e6a9f2" (UID: "9563448c-19ba-4eb3-9ede-9429e5e6a9f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:31:30.273366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.256826    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:30.283314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.281232    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:30.283314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.281270    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:30.283314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.281336    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:30.283314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.281353    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:30.283314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.281370    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:30.302577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.289055    4851 scope.go:122] "RemoveContainer" containerID="03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee"
Nov 02 01:31:30.302577 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:31:30.289807    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee\": container with ID starting with 03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee not found: ID does not exist" containerID="03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee"
Nov 02 01:31:30.302577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.289850    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee"} err="failed to get container status \"03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee\": rpc error: code = NotFound desc = could not find container \"03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee\": container with ID starting with 03f275b66902ce8fed036c01d3252208a2b7fbf82ab8399a5aac0981a7e526ee not found: ID does not exist"
Nov 02 01:31:30.302577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.297128    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-envoy" pod "9563448c-19ba-4eb3-9ede-9429e5e6a9f2" (UID: "9563448c-19ba-4eb3-9ede-9429e5e6a9f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:31:30.302577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.297334    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-token" pod "9563448c-19ba-4eb3-9ede-9429e5e6a9f2" (UID: "9563448c-19ba-4eb3-9ede-9429e5e6a9f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:30.302577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.302484    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-podinfo" pod "9563448c-19ba-4eb3-9ede-9429e5e6a9f2" (UID: "9563448c-19ba-4eb3-9ede-9429e5e6a9f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:31:30.303817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.303796    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-kube-api-access-jxkx8" pod "9563448c-19ba-4eb3-9ede-9429e5e6a9f2" (UID: "9563448c-19ba-4eb3-9ede-9429e5e6a9f2"). InnerVolumeSpecName "kube-api-access-jxkx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:30.383009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.382978    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxkx8\" (UniqueName: \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-kube-api-access-jxkx8\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:30.383581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.383014    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:30.383581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.383030    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:30.383581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.383044    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9563448c-19ba-4eb3-9ede-9429e5e6a9f2-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:30.404916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.404884    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"]
Nov 02 01:31:30.412079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.412051    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6c45ad5e-openshift-default-f6956fc-m7lgg"]
Nov 02 01:31:30.868773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.868738    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9563448c-19ba-4eb3-9ede-9429e5e6a9f2" path="/var/lib/kubelet/pods/9563448c-19ba-4eb3-9ede-9429e5e6a9f2/volumes"
Nov 02 01:31:30.913677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:30.913613    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:31.085862 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:31.085778    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" event={"ID":"83e32e09-9992-4e33-906c-cb088eb20ddc","Type":"ContainerStarted","Data":"6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3"}
Nov 02 01:31:31.775317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:31.768829    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:32.045379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:32.036610    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:31:32.045379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:32.036722    4851 patch_prober.go:29] interesting pod/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 01:31:32.045379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:32.036763    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 01:31:32.526963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:32.526925    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:33.054316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:33.052733    4851 patch_prober.go:29] interesting pod/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 01:31:33.054316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:33.052799    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 01:31:33.175589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:33.175543    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:33.907281 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:33.907218    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:34.035318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:34.033885    4851 patch_prober.go:29] interesting pod/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 01:31:34.035318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:34.033945    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 01:31:34.700889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:34.700847    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:35.034837 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.034746    4851 patch_prober.go:29] interesting pod/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 01:31:35.035475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.035443    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 01:31:35.500554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.500512    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"]
Nov 02 01:31:35.501356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.501334    4851 cpu_manager.go:414] "containerMap: removing container" podUID="471fa604-d12d-44d4-878f-8d6138a47835" containerName="revoke-keys"
Nov 02 01:31:35.501492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.501357    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="471fa604-d12d-44d4-878f-8d6138a47835" containerName="revoke-keys"
Nov 02 01:31:35.501492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.501432    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9563448c-19ba-4eb3-9ede-9429e5e6a9f2" containerName="istio-proxy"
Nov 02 01:31:35.501492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.501441    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9563448c-19ba-4eb3-9ede-9429e5e6a9f2" containerName="istio-proxy"
Nov 02 01:31:35.501659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.501626    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9563448c-19ba-4eb3-9ede-9429e5e6a9f2" containerName="istio-proxy"
Nov 02 01:31:35.501659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.501648    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="471fa604-d12d-44d4-878f-8d6138a47835" containerName="revoke-keys"
Nov 02 01:31:35.505675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.505651    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:35.513434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.513401    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:31:35.515141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.515116    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:31:35.525316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.525261    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"]
Nov 02 01:31:35.527743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.527716    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:31:35.527932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.527843    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:31:35.591379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.588094    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9kb\" (UniqueName: \"kubernetes.io/projected/2fecebf4-e5f8-479c-b6b3-f1230f698b48-kube-api-access-5d9kb\") pod \"maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:35.591379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.588354    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:35.595504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.592535    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:35.609279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.609232    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:35.735301 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.735249    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:35.771774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.771689    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9kb\" (UniqueName: \"kubernetes.io/projected/2fecebf4-e5f8-479c-b6b3-f1230f698b48-kube-api-access-5d9kb\") pod \"maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:35.772267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.772236    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:35.785331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.773457    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:35.856484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.835926    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:35.899757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.885829    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9kb\" (UniqueName: \"kubernetes.io/projected/2fecebf4-e5f8-479c-b6b3-f1230f698b48-kube-api-access-5d9kb\") pod \"maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:35.932329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:35.918087    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:36.039824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:36.039788    4851 patch_prober.go:29] interesting pod/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 01:31:36.040411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:36.040363    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 01:31:36.326420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:36.306208    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:36.691313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:36.674207    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:37.125621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:37.125535    4851 patch_prober.go:29] interesting pod/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 01:31:37.126148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:37.126117    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 01:31:37.177326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:37.177138    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"]
Nov 02 01:31:37.330785 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:31:37.330753    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fecebf4_e5f8_479c_b6b3_f1230f698b48.slice/crio-2af9934c4f1daf6eed6e772537a1e4f40cb87b505e0319aa8be935b348cd6ddd WatchSource:0}: Error finding container 2af9934c4f1daf6eed6e772537a1e4f40cb87b505e0319aa8be935b348cd6ddd: Status 404 returned error can't find the container with id 2af9934c4f1daf6eed6e772537a1e4f40cb87b505e0319aa8be935b348cd6ddd
Nov 02 01:31:37.685317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:37.667929    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:37.711318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:37.711012    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:37.716324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:37.711494    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:37.740313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:37.738745    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g" event={"ID":"2fecebf4-e5f8-479c-b6b3-f1230f698b48","Type":"ContainerStarted","Data":"2af9934c4f1daf6eed6e772537a1e4f40cb87b505e0319aa8be935b348cd6ddd"}
Nov 02 01:31:37.906315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:37.905365    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"]
Nov 02 01:31:37.974657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:37.974610    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:38.151328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.151262    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"]
Nov 02 01:31:38.151848 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.151537    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:31:38.162397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.152253    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:38.172493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.162856    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:31:38.172493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.164541    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:31:38.268945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.268898    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vmp\" (UniqueName: \"kubernetes.io/projected/767f5dac-be5a-449c-a99c-33efcb91c1e7-kube-api-access-p4vmp\") pod \"maas-api-revoke-keys-e2e-ait-89833f31-48jrf\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:38.269316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.269266    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-89833f31-48jrf\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:38.269598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.269577    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-89833f31-48jrf\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:38.287860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.287823    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:31:38.396461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.396396    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:31:38.468020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.398850    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-89833f31-48jrf\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:38.468692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.468652    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vmp\" (UniqueName: \"kubernetes.io/projected/767f5dac-be5a-449c-a99c-33efcb91c1e7-kube-api-access-p4vmp\") pod \"maas-api-revoke-keys-e2e-ait-89833f31-48jrf\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:38.468997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.468974    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-89833f31-48jrf\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:38.470115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.470093    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-89833f31-48jrf\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:38.489319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.477749    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:38.514811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.494153    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-89833f31-48jrf\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:38.540317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.507321    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:38.594436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.594386    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:38.652791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.649789    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vmp\" (UniqueName: \"kubernetes.io/projected/767f5dac-be5a-449c-a99c-33efcb91c1e7-kube-api-access-p4vmp\") pod \"maas-api-revoke-keys-e2e-ait-89833f31-48jrf\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:38.652791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.651385    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:31:38.693587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.677104    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:38.695496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.695430    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:31:38.726320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.696306    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:31:38.726320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.711168    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:38.726320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.713224    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:31:38.726320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.713305    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:31:38.793484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.791762    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:38.793484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.792363    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:31:38.930327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:38.929275    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:39.038316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.022440    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:39.298319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.296149    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:39.337494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.337439    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:39.540519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.540439    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:39.600389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.599797    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:31:39.711505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.709014    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:39.718596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.717486    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:39.722661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.722611    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:39.750545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.750510    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:31:39.750923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.750904    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:31:39.751514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.751496    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:31:39.751714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.751701    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:31:39.751890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.751877    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:31:39.752056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.752042    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:31:39.752210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.752197    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:31:39.752392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.752377    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:31:39.752566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.752553    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:31:39.752733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.752719    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:31:39.894890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.867797    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:39.932234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:39.932086    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:40.080494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:40.080428    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:31:40.140321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:40.140272    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:31:40.203692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:40.198276    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:31:40.284723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:40.283991    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:31:40.319523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:40.318921    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:31:40.381601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:40.381559    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:31:40.429903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:40.429863    4851 generic.go:356] "Generic (PLEG): container finished" podID="2fecebf4-e5f8-479c-b6b3-f1230f698b48" containerID="0db41ee6e0eadfdd369f523e94bc5fec0e4f36b22b5ec693f518b83fe29b8f4f" exitCode=0
Nov 02 01:31:40.430220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:40.430195    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g" event={"ID":"2fecebf4-e5f8-479c-b6b3-f1230f698b48","Type":"ContainerDied","Data":"0db41ee6e0eadfdd369f523e94bc5fec0e4f36b22b5ec693f518b83fe29b8f4f"}
Nov 02 01:31:40.518750 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:31:40.517724    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767f5dac_be5a_449c_a99c_33efcb91c1e7.slice/crio-925cb704536c62c73e77972699c78f99e1c12f521266bdfcaa5c865dbde41ce2 WatchSource:0}: Error finding container 925cb704536c62c73e77972699c78f99e1c12f521266bdfcaa5c865dbde41ce2: Status 404 returned error can't find the container with id 925cb704536c62c73e77972699c78f99e1c12f521266bdfcaa5c865dbde41ce2
Nov 02 01:31:40.535586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:40.535268    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"]
Nov 02 01:31:40.805506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:40.805381    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:41.465609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.465571    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf" event={"ID":"767f5dac-be5a-449c-a99c-33efcb91c1e7","Type":"ContainerStarted","Data":"35e084fbb0cb99f30bf3b88fbc293be2fcf4ff8b67f273dc84c2053ccfbe8f7f"}
Nov 02 01:31:41.466016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.465633    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf" event={"ID":"767f5dac-be5a-449c-a99c-33efcb91c1e7","Type":"ContainerStarted","Data":"925cb704536c62c73e77972699c78f99e1c12f521266bdfcaa5c865dbde41ce2"}
Nov 02 01:31:41.691406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.691377    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:41.713283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.713217    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf" podStartSLOduration=4.7131983680000005 podStartE2EDuration="4.713198368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:41.485129554 +0000 UTC m=+1006.790062380" watchObservedRunningTime="2026-11-02 01:31:41.713198368 +0000 UTC m=+1007.018131195"
Nov 02 01:31:41.719067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.718994    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:41.834595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.829952    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fecebf4-e5f8-479c-b6b3-f1230f698b48-kube-api-access-5d9kb\" (UniqueName: \"kubernetes.io/projected/2fecebf4-e5f8-479c-b6b3-f1230f698b48-kube-api-access-5d9kb\") pod \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") "
Nov 02 01:31:41.834595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.830282    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-xks-ca\") pod \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") "
Nov 02 01:31:41.834595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.830392    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-openshift-ca\") pod \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\" (UID: \"2fecebf4-e5f8-479c-b6b3-f1230f698b48\") "
Nov 02 01:31:41.834595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.831497    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-openshift-ca" pod "2fecebf4-e5f8-479c-b6b3-f1230f698b48" (UID: "2fecebf4-e5f8-479c-b6b3-f1230f698b48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:31:41.932456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.931200    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fecebf4-e5f8-479c-b6b3-f1230f698b48-kube-api-access-5d9kb" pod "2fecebf4-e5f8-479c-b6b3-f1230f698b48" (UID: "2fecebf4-e5f8-479c-b6b3-f1230f698b48"). InnerVolumeSpecName "kube-api-access-5d9kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:41.932688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.932659    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-xks-ca" pod "2fecebf4-e5f8-479c-b6b3-f1230f698b48" (UID: "2fecebf4-e5f8-479c-b6b3-f1230f698b48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:31:41.947691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.947653    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:41.947913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.947895    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fecebf4-e5f8-479c-b6b3-f1230f698b48-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:41.950240 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:41.950218    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9kb\" (UniqueName: \"kubernetes.io/projected/2fecebf4-e5f8-479c-b6b3-f1230f698b48-kube-api-access-5d9kb\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:42.479807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:42.479769    4851 generic.go:356] "Generic (PLEG): container finished" podID="767f5dac-be5a-449c-a99c-33efcb91c1e7" containerID="35e084fbb0cb99f30bf3b88fbc293be2fcf4ff8b67f273dc84c2053ccfbe8f7f" exitCode=0
Nov 02 01:31:42.480248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:42.479861    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf" event={"ID":"767f5dac-be5a-449c-a99c-33efcb91c1e7","Type":"ContainerDied","Data":"35e084fbb0cb99f30bf3b88fbc293be2fcf4ff8b67f273dc84c2053ccfbe8f7f"}
Nov 02 01:31:42.481684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:42.481664    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"
Nov 02 01:31:42.481684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:42.481674    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g" event={"ID":"2fecebf4-e5f8-479c-b6b3-f1230f698b48","Type":"ContainerDied","Data":"2af9934c4f1daf6eed6e772537a1e4f40cb87b505e0319aa8be935b348cd6ddd"}
Nov 02 01:31:42.481835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:42.481714    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2af9934c4f1daf6eed6e772537a1e4f40cb87b505e0319aa8be935b348cd6ddd"
Nov 02 01:31:42.508273 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:42.508240    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:43.291091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.291058    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:43.672948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.672922    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:43.797041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.797001    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-xks-ca\") pod \"767f5dac-be5a-449c-a99c-33efcb91c1e7\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") "
Nov 02 01:31:43.797211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.797170    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-openshift-ca\") pod \"767f5dac-be5a-449c-a99c-33efcb91c1e7\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") "
Nov 02 01:31:43.797468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.797432    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767f5dac-be5a-449c-a99c-33efcb91c1e7-kube-api-access-p4vmp\" (UniqueName: \"kubernetes.io/projected/767f5dac-be5a-449c-a99c-33efcb91c1e7-kube-api-access-p4vmp\") pod \"767f5dac-be5a-449c-a99c-33efcb91c1e7\" (UID: \"767f5dac-be5a-449c-a99c-33efcb91c1e7\") "
Nov 02 01:31:43.797654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.797587    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-openshift-ca" pod "767f5dac-be5a-449c-a99c-33efcb91c1e7" (UID: "767f5dac-be5a-449c-a99c-33efcb91c1e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:31:43.798452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.798422    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:43.799658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.799628    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-xks-ca" pod "767f5dac-be5a-449c-a99c-33efcb91c1e7" (UID: "767f5dac-be5a-449c-a99c-33efcb91c1e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:31:43.799995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.799976    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767f5dac-be5a-449c-a99c-33efcb91c1e7-kube-api-access-p4vmp" pod "767f5dac-be5a-449c-a99c-33efcb91c1e7" (UID: "767f5dac-be5a-449c-a99c-33efcb91c1e7"). InnerVolumeSpecName "kube-api-access-p4vmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:31:43.900409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.900362    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4vmp\" (UniqueName: \"kubernetes.io/projected/767f5dac-be5a-449c-a99c-33efcb91c1e7-kube-api-access-p4vmp\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:43.900409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:43.900411    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767f5dac-be5a-449c-a99c-33efcb91c1e7-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:31:44.111331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:44.111212    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:44.501920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:44.501892    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"
Nov 02 01:31:44.502129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:44.501928    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf" event={"ID":"767f5dac-be5a-449c-a99c-33efcb91c1e7","Type":"ContainerDied","Data":"925cb704536c62c73e77972699c78f99e1c12f521266bdfcaa5c865dbde41ce2"}
Nov 02 01:31:44.502129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:44.501983    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="925cb704536c62c73e77972699c78f99e1c12f521266bdfcaa5c865dbde41ce2"
Nov 02 01:31:44.806463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:44.806370    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:45.576145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:45.574246    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:45.826441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:45.826356    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:45.959940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:45.959899    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:46.436358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:46.436321    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:46.565503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:46.565468    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:47.230498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:47.230469    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:47.576493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:47.576462    4851 generic.go:356] "Generic (PLEG): container finished" podID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerID="dccd95306e86e67e44813997b9aff23ca1fa1bf1f48a756bb5bf3a033079dc60" exitCode=0
Nov 02 01:31:47.576714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:47.576535    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" event={"ID":"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6","Type":"ContainerDied","Data":"dccd95306e86e67e44813997b9aff23ca1fa1bf1f48a756bb5bf3a033079dc60"}
Nov 02 01:31:47.577083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:47.577064    4851 scope.go:122] "RemoveContainer" containerID="dccd95306e86e67e44813997b9aff23ca1fa1bf1f48a756bb5bf3a033079dc60"
Nov 02 01:31:47.578855 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:47.578827    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:47.578927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:47.578895    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:47.578966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:47.578928    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:47.674469 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:31:47.674440    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:31:48.146783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:48.146427    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:48.272945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:48.272906    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:48.646023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:48.645938    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" event={"ID":"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6","Type":"ContainerStarted","Data":"43e9fb79292895a1f22c3323d03b83632649d7c1d60d14ebf602b35d968d0a59"}
Nov 02 01:31:48.825788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:48.825747    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:49.049315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:49.038611    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:49.072313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:49.071772    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:49.072313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:49.071910    4851 patch_prober.go:29] interesting pod/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 01:31:49.072313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:49.071952    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 01:31:49.252036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:49.251718    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:49.286744 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:49.286693    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"]
Nov 02 01:31:49.294941 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:49.294914    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9a278165-gjd6g"]
Nov 02 01:31:49.445319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:49.444874    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:49.776910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:49.612232    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:50.069182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:50.069092    4851 patch_prober.go:29] interesting pod/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 01:31:50.069182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:50.069146    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 01:31:50.366281 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:50.366190    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:50.614943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:50.609702    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:50.950442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:50.943008    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:51.057312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:51.014141    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fecebf4-e5f8-479c-b6b3-f1230f698b48" path="/var/lib/kubelet/pods/2fecebf4-e5f8-479c-b6b3-f1230f698b48/volumes"
Nov 02 01:31:51.088389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:51.087374    4851 patch_prober.go:29] interesting pod/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 01:31:51.142161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:51.142105    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 01:31:51.184846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:51.184701    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:51.185355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:51.185273    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"]
Nov 02 01:31:51.280492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:51.244568    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89833f31-48jrf"]
Nov 02 01:31:51.573774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:51.573583    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"]
Nov 02 01:31:51.574259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:51.573777    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerName="istio-proxy" containerID="cri-o://43e9fb79292895a1f22c3323d03b83632649d7c1d60d14ebf602b35d968d0a59" gracePeriod=30
Nov 02 01:31:51.667957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:51.667875    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:52.254312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.253996    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:52.424314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.423863    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:52.771027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.770996    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"]
Nov 02 01:31:52.771875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.771852    4851 cpu_manager.go:414] "containerMap: removing container" podUID="2fecebf4-e5f8-479c-b6b3-f1230f698b48" containerName="revoke-keys"
Nov 02 01:31:52.771875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.771873    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fecebf4-e5f8-479c-b6b3-f1230f698b48" containerName="revoke-keys"
Nov 02 01:31:52.772059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.771905    4851 cpu_manager.go:414] "containerMap: removing container" podUID="767f5dac-be5a-449c-a99c-33efcb91c1e7" containerName="revoke-keys"
Nov 02 01:31:52.772059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.771914    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767f5dac-be5a-449c-a99c-33efcb91c1e7" containerName="revoke-keys"
Nov 02 01:31:52.772142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.772064    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fecebf4-e5f8-479c-b6b3-f1230f698b48" containerName="revoke-keys"
Nov 02 01:31:52.772142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.772084    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="767f5dac-be5a-449c-a99c-33efcb91c1e7" containerName="revoke-keys"
Nov 02 01:31:52.776419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.776399    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:52.790139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.790112    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-eeeb1a-openshift-default-dockercfg-hw5g9\""
Nov 02 01:31:52.795329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:52.795272    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"]
Nov 02 01:31:53.014316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.007389    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:53.096373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.084839    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b253b7-6535-426d-814c-9c509d7c2992-istiod-ca-cert\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.096373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.084908    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b253b7-6535-426d-814c-9c509d7c2992-istio-podinfo\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.096373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.085041    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-socket\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.096373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.085093    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxwc6\" (UniqueName: \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-kube-api-access-qxwc6\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.096373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.085138    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-data\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.096373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.085183    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-certs\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.096373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.085238    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-credential-socket\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.096373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.085337    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-envoy\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.096373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.085640    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-istio-token\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.176313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.175644    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767f5dac-be5a-449c-a99c-33efcb91c1e7" path="/var/lib/kubelet/pods/767f5dac-be5a-449c-a99c-33efcb91c1e7/volumes"
Nov 02 01:31:53.192371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.188620    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-istio-token\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.192371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.188696    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b253b7-6535-426d-814c-9c509d7c2992-istiod-ca-cert\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.192371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.188740    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b253b7-6535-426d-814c-9c509d7c2992-istio-podinfo\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.192371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.188823    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-socket\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.192371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.188881    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxwc6\" (UniqueName: \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-kube-api-access-qxwc6\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.192371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.188934    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-data\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.192371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.188989    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-certs\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.192371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.189059    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-credential-socket\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.192371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.189153    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-envoy\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.195656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.194846    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-data\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.195656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.195139    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-socket\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.197092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.196041    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-certs\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.197092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.196696    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b253b7-6535-426d-814c-9c509d7c2992-istiod-ca-cert\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.197092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.196942    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-credential-socket\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.213550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.213474    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b253b7-6535-426d-814c-9c509d7c2992-istio-podinfo\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.215136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.215017    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:53.223818 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.221577    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-istio-token\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.224789 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.224720    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxwc6\" (UniqueName: \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-kube-api-access-qxwc6\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.333280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.333248    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-envoy\") pod \"e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") " pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.430756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.429399    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:53.460822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:53.448700    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:54.005920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:54.005890    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:54.006681 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:31:54.006653    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b253b7_6535_426d_814c_9c509d7c2992.slice/crio-49f4fe0ad5365351ba026094acdade8bf37ed8de20cc8cc8a1d36dc5c6e310a9 WatchSource:0}: Error finding container 49f4fe0ad5365351ba026094acdade8bf37ed8de20cc8cc8a1d36dc5c6e310a9: Status 404 returned error can't find the container with id 49f4fe0ad5365351ba026094acdade8bf37ed8de20cc8cc8a1d36dc5c6e310a9
Nov 02 01:31:54.008772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:54.008747    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"]
Nov 02 01:31:54.009463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:54.009436    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:54.009516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:54.009499    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:54.009561 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:54.009527    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:54.396111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:54.396065    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" event={"ID":"20b253b7-6535-426d-814c-9c509d7c2992","Type":"ContainerStarted","Data":"40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36"}
Nov 02 01:31:54.396631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:54.396184    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" event={"ID":"20b253b7-6535-426d-814c-9c509d7c2992","Type":"ContainerStarted","Data":"49f4fe0ad5365351ba026094acdade8bf37ed8de20cc8cc8a1d36dc5c6e310a9"}
Nov 02 01:31:54.425186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:54.425125    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" podStartSLOduration=2.425110391 podStartE2EDuration="2.425110391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:54.422912857 +0000 UTC m=+1019.727845685" watchObservedRunningTime="2026-11-02 01:31:54.425110391 +0000 UTC m=+1019.730043217"
Nov 02 01:31:54.430137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:54.430110    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:54.776065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:54.776032    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:55.437229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:55.437200    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:55.539241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:55.539208    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:55.899381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:55.899350    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:31:56.331892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:56.331854    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:56.458756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:56.458724    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:56.462626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:56.462598    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:31:56.646008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:56.645929    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:56.790095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:56.790064    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:57.132030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:57.131994    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:57.837278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:57.837242    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:57.985276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:57.985242    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:58.784856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:58.784815    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:59.037977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:59.037899    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"]
Nov 02 01:31:59.043863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:59.043842    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"]
Nov 02 01:31:59.049068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:59.049048    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e0ad96-xpjhk"]
Nov 02 01:31:59.054128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:59.054106    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6dc38c-lpnsh"]
Nov 02 01:31:59.529153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:31:59.529120    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:00.026054 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:00.026020    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"]
Nov 02 01:32:00.031879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:00.031852    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1628473-m7q9j"]
Nov 02 01:32:00.219708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:00.219679    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:00.462507 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:32:00.462461    4851 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab8b8cd_bb53_4b88_a82e_6b33f75708a6.slice/crio-conmon-43e9fb79292895a1f22c3323d03b83632649d7c1d60d14ebf602b35d968d0a59.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab8b8cd_bb53_4b88_a82e_6b33f75708a6.slice/crio-43e9fb79292895a1f22c3323d03b83632649d7c1d60d14ebf602b35d968d0a59.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 01:32:00.501435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:00.501402    4851 generic.go:356] "Generic (PLEG): container finished" podID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerID="43e9fb79292895a1f22c3323d03b83632649d7c1d60d14ebf602b35d968d0a59" exitCode=0
Nov 02 01:32:00.501619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:00.501530    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" event={"ID":"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6","Type":"ContainerDied","Data":"43e9fb79292895a1f22c3323d03b83632649d7c1d60d14ebf602b35d968d0a59"}
Nov 02 01:32:00.501666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:00.501630    4851 scope.go:122] "RemoveContainer" containerID="dccd95306e86e67e44813997b9aff23ca1fa1bf1f48a756bb5bf3a033079dc60"
Nov 02 01:32:00.849880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:00.849846    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d78919-d93b-4a18-8dca-f5034b63d756" path="/var/lib/kubelet/pods/80d78919-d93b-4a18-8dca-f5034b63d756/volumes"
Nov 02 01:32:00.850385 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:00.850359    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b547c344-04eb-4df1-80c1-575bf46d3f3d" path="/var/lib/kubelet/pods/b547c344-04eb-4df1-80c1-575bf46d3f3d/volumes"
Nov 02 01:32:00.850855 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:00.850838    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2e8e49-e371-4d51-88e8-3fe8e3b3ea51" path="/var/lib/kubelet/pods/df2e8e49-e371-4d51-88e8-3fe8e3b3ea51/volumes"
Nov 02 01:32:00.918768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:00.918735    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:01.104942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.104902    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:32:01.248694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.248663    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-token\" (UniqueName: \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-token\") pod \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") "
Nov 02 01:32:01.249179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.248745    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-credential-socket\") pod \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") "
Nov 02 01:32:01.249179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.248840    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istiod-ca-cert\") pod \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") "
Nov 02 01:32:01.249179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.248904    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-podinfo\") pod \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") "
Nov 02 01:32:01.249179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.249044    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-kube-api-access-vbjhd\" (UniqueName: \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-kube-api-access-vbjhd\") pod \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") "
Nov 02 01:32:01.249179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.249094    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-data\") pod \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") "
Nov 02 01:32:01.249179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.249093    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-credential-socket" pod "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" (UID: "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:32:01.249532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.249213    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-socket\") pod \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") "
Nov 02 01:32:01.249532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.249250    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istiod-ca-cert" pod "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" (UID: "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:01.249532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.249328    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-envoy\") pod \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") "
Nov 02 01:32:01.249532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.249377    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-certs\") pod \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\" (UID: \"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6\") "
Nov 02 01:32:01.249532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.249416    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-data" pod "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" (UID: "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:32:01.249825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.249800    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-socket" pod "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" (UID: "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:32:01.249898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.249845    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-certs" pod "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" (UID: "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:32:01.250092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.250051    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:01.250092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.250077    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:01.250092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.250092    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:01.250598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.250106    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:01.250598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.250120    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:01.251964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.251902    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-podinfo" pod "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" (UID: "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:32:01.252210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.252188    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-envoy" pod "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" (UID: "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:32:01.252360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.252337    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-kube-api-access-vbjhd" pod "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" (UID: "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6"). InnerVolumeSpecName "kube-api-access-vbjhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:01.252455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.252433    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-token" pod "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" (UID: "8ab8b8cd-bb53-4b88-a82e-6b33f75708a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:01.351532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.351452    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbjhd\" (UniqueName: \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-kube-api-access-vbjhd\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:01.351532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.351480    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:01.351532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.351492    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:01.351532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.351501    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:01.513141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.513104    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"
Nov 02 01:32:01.513141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.513126    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc" event={"ID":"8ab8b8cd-bb53-4b88-a82e-6b33f75708a6","Type":"ContainerDied","Data":"dfd9f0421e267cb0a3adc32e9dd4f46e9b046fcdae77989f06404220f4290aca"}
Nov 02 01:32:01.513445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.513198    4851 scope.go:122] "RemoveContainer" containerID="43e9fb79292895a1f22c3323d03b83632649d7c1d60d14ebf602b35d968d0a59"
Nov 02 01:32:01.555684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.555645    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"]
Nov 02 01:32:01.561952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.561929    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-89833f31-openshift-default-7f98ff894c-8mgqc"]
Nov 02 01:32:01.815456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:01.812551    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:02.376696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:02.376659    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:02.848955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:02.848922    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" path="/var/lib/kubelet/pods/8ab8b8cd-bb53-4b88-a82e-6b33f75708a6/volumes"
Nov 02 01:32:03.041424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:03.041389    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"]
Nov 02 01:32:03.052577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:03.052540    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5ec0c2-z8rkv"]
Nov 02 01:32:03.102053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:03.101947    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:03.236339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:03.236281    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:03.427175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:03.427147    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:03.952951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:03.952919    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:04.608023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:04.607985    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:04.848593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:04.848557    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53319cf-e568-412f-be8c-cd5b056ff07e" path="/var/lib/kubelet/pods/a53319cf-e568-412f-be8c-cd5b056ff07e/volumes"
Nov 02 01:32:05.365020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:05.361792    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:06.038601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:06.038571    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:06.727366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:06.727333    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:07.421348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:07.421314    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:08.311075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:08.311036    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:09.022415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:09.021930    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:09.033252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:09.033218    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"]
Nov 02 01:32:09.081326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:09.079782    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc15bc0f-ldgz8"]
Nov 02 01:32:10.013883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:10.010122    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:10.920229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:10.920189    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9645b072-7e8a-4f87-8961-704dc4b65345" path="/var/lib/kubelet/pods/9645b072-7e8a-4f87-8961-704dc4b65345/volumes"
Nov 02 01:32:11.015430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:11.015397    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:11.077314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:11.074370    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"]
Nov 02 01:32:11.214373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:11.214267    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"]
Nov 02 01:32:11.245323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:11.234349    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a39108-bsjcb"]
Nov 02 01:32:11.245323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:11.238266    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1442a92-pf9wx"]
Nov 02 01:32:11.911318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:11.910380    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:12.814950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:12.814909    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:12.909330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:12.907999    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2" path="/var/lib/kubelet/pods/2d3272a2-71ff-4a53-8c0e-3fbadf0a8fb2/volumes"
Nov 02 01:32:12.909330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:12.908911    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e8246b-bbb1-4bec-8fb5-c56925b25887" path="/var/lib/kubelet/pods/62e8246b-bbb1-4bec-8fb5-c56925b25887/volumes"
Nov 02 01:32:13.604316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:13.604255    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:14.312914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:14.312872    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:14.959118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:14.959081    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:15.064942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:15.064879    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"]
Nov 02 01:32:15.100763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:15.100730    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8416d186-wxb9w"]
Nov 02 01:32:15.629930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:15.629884    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:16.296179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:16.296145    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:16.849118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:16.849088    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d323372e-2956-4b58-9e74-dffbce95381c" path="/var/lib/kubelet/pods/d323372e-2956-4b58-9e74-dffbce95381c/volumes"
Nov 02 01:32:16.857942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:16.857911    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:17.024883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:17.024848    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:17.027144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:17.027114    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"]
Nov 02 01:32:17.062034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:17.061993    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e25e0d-bqw7g"]
Nov 02 01:32:17.367724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:17.367681    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:17.659557 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:32:17.659479    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:32:17.822820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:17.822787    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:17.954253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:17.954215    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:18.742197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:18.742154    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:18.897664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:18.897629    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f38900d-daa1-4af7-be67-1ccab966f772" path="/var/lib/kubelet/pods/5f38900d-daa1-4af7-be67-1ccab966f772/volumes"
Nov 02 01:32:19.520985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:19.520951    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:20.230506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:20.230470    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:20.982991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:20.982950    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:21.811784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:21.811753    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:22.085901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:22.085826    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:22.219748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:22.219718    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:22.720011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:22.719976    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:23.044436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:23.044356    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"]
Nov 02 01:32:23.050124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:23.050091    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"]
Nov 02 01:32:23.065730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:23.065673    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19e19298-8jfds"]
Nov 02 01:32:23.077557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:23.077525    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae11ef5-zsb8x"]
Nov 02 01:32:23.121402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:23.121366    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:23.304447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:23.284201    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:23.742973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:23.742934    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:24.026876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:24.026799    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"]
Nov 02 01:32:24.032126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:24.032104    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93c99148-2hn7m"]
Nov 02 01:32:24.523032 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:24.522995    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:24.883088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:24.883007    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1994b68d-08ae-4227-bc29-97c19358fec3" path="/var/lib/kubelet/pods/1994b68d-08ae-4227-bc29-97c19358fec3/volumes"
Nov 02 01:32:24.883592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:24.883574    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e5ad66-39b3-44b0-9f51-5deb62085a2b" path="/var/lib/kubelet/pods/53e5ad66-39b3-44b0-9f51-5deb62085a2b/volumes"
Nov 02 01:32:24.883992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:24.883977    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeaa1973-6b38-4517-ac80-9670e3a42655" path="/var/lib/kubelet/pods/aeaa1973-6b38-4517-ac80-9670e3a42655/volumes"
Nov 02 01:32:25.289104 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:25.289061    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:25.930887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:25.930851    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:26.055538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:26.055505    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:26.324127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:26.323984    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:26.839455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:26.839428    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:27.031843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:27.031808    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"]
Nov 02 01:32:27.037691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:27.037659    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f32e8a0a-2j8nz"]
Nov 02 01:32:27.722003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:27.721965    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:28.026997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:28.026911    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"]
Nov 02 01:32:28.032244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:28.032220    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e24150c6-xcvnh"]
Nov 02 01:32:28.416977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:28.416940    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:28.848041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:28.847967    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42bf9d27-9ccc-43cf-bc68-9e6987378520" path="/var/lib/kubelet/pods/42bf9d27-9ccc-43cf-bc68-9e6987378520/volumes"
Nov 02 01:32:28.848508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:28.848494    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8919b179-eceb-4b01-9a7a-2a56983be518" path="/var/lib/kubelet/pods/8919b179-eceb-4b01-9a7a-2a56983be518/volumes"
Nov 02 01:32:29.187166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:29.187126    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:29.339316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:29.338911    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:30.099567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:30.099521    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:30.875991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:30.875959    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:31.711863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:31.711824    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:32.419847 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:32.419784    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:32.703310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:32.703219    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:32.838976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:32.838938    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:33.105198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:33.105116    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:33.807535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:33.807497    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:34.029025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:34.028988    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"]
Nov 02 01:32:34.034817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:34.034794    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d85c94-jsfp2"]
Nov 02 01:32:34.510649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:34.510615    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:34.849543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:34.849464    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ed7c1d-eab3-421c-940e-328af9afc7f6" path="/var/lib/kubelet/pods/b5ed7c1d-eab3-421c-940e-328af9afc7f6/volumes"
Nov 02 01:32:35.379955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:35.379923    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:36.027924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.027886    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"]
Nov 02 01:32:36.033126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.033106    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7aec4bf-x92wk"]
Nov 02 01:32:36.107710 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.107683    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:36.243248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.239770    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:36.434736 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.434701    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:36.490016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.489983    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"]
Nov 02 01:32:36.491448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.491418    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerName="istio-proxy"
Nov 02 01:32:36.491625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.491602    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerName="istio-proxy"
Nov 02 01:32:36.491759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.491639    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerName="istio-proxy"
Nov 02 01:32:36.491759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.491650    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerName="istio-proxy"
Nov 02 01:32:36.491889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.491845    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerName="istio-proxy"
Nov 02 01:32:36.496727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.496708    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.503080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.503059    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-50537f4f-openshift-default-dockercfg-2fm44\""
Nov 02 01:32:36.508851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.508828    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"]
Nov 02 01:32:36.580435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.580407    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-certs\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.580605 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.580562    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-istio-token\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.580653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.580633    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb40876a-6ba0-40ee-864a-103d087b918e-istiod-ca-cert\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.580700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.580677    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.580743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.580701    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-data\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.580799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.580777    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb40876a-6ba0-40ee-864a-103d087b918e-istio-podinfo\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.580842 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.580834    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-credential-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.580942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.580929    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-envoy\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.580985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.580950    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2t4j\" (UniqueName: \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-kube-api-access-b2t4j\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.682174    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-envoy\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.682223    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2t4j\" (UniqueName: \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-kube-api-access-b2t4j\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.682317    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-certs\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.682486    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-istio-token\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.682572    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb40876a-6ba0-40ee-864a-103d087b918e-istiod-ca-cert\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.682640    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.682676    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-data\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.682755    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb40876a-6ba0-40ee-864a-103d087b918e-istio-podinfo\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.682857    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-credential-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.683438    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-credential-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.689369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.686545    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:36.718222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.698503    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb40876a-6ba0-40ee-864a-103d087b918e-istiod-ca-cert\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.718222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.702676    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-envoy\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.718222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.702985    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.718222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.703249    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-data\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.718222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.709804    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-certs\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.751064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.751034    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2t4j\" (UniqueName: \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-kube-api-access-b2t4j\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.751851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.751782    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-istio-token\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.830315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.823644    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb40876a-6ba0-40ee-864a-103d087b918e-istio-podinfo\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.830315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.823962    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:36.939473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.939389    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:36.971694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:36.971659    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339aa318-ced7-44f5-929c-09fa6493e1fd" path="/var/lib/kubelet/pods/339aa318-ced7-44f5-929c-09fa6493e1fd/volumes"
Nov 02 01:32:37.076365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.076317    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:37.155965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.155936    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"]
Nov 02 01:32:37.156978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.156957    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab8b8cd-bb53-4b88-a82e-6b33f75708a6" containerName="istio-proxy"
Nov 02 01:32:37.162268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.162248    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.167238 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.167211    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:32:37.167366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.167333    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:32:37.171487 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.171371    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"]
Nov 02 01:32:37.189651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.189557    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:32:37.189651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.189568    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:32:37.209520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.209483    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:37.271318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.241575    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-616e3ed7-sf646\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.271318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.241668    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8sj\" (UniqueName: \"kubernetes.io/projected/760cd3dd-8bdf-471a-a9a1-24518be5643a-kube-api-access-2h8sj\") pod \"maas-api-revoke-keys-e2e-mt-616e3ed7-sf646\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.271318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.241849    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-616e3ed7-sf646\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.368376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.354528    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-616e3ed7-sf646\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.369407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.369379    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8sj\" (UniqueName: \"kubernetes.io/projected/760cd3dd-8bdf-471a-a9a1-24518be5643a-kube-api-access-2h8sj\") pod \"maas-api-revoke-keys-e2e-mt-616e3ed7-sf646\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.369719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.369699    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-616e3ed7-sf646\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.370585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.370476    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-616e3ed7-sf646\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.372480 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:32:37.371430    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb40876a_6ba0_40ee_864a_103d087b918e.slice/crio-cf4245ba5461575f8f358469e0a61e4105efb708fe34dc85a4cd169c9f2eb3ee WatchSource:0}: Error finding container cf4245ba5461575f8f358469e0a61e4105efb708fe34dc85a4cd169c9f2eb3ee: Status 404 returned error can't find the container with id cf4245ba5461575f8f358469e0a61e4105efb708fe34dc85a4cd169c9f2eb3ee
Nov 02 01:32:37.383858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.383824    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:32:37.383986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.383892    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-616e3ed7-sf646\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.383986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.383910    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:32:37.383986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.383959    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:32:37.385433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.385370    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8sj\" (UniqueName: \"kubernetes.io/projected/760cd3dd-8bdf-471a-a9a1-24518be5643a-kube-api-access-2h8sj\") pod \"maas-api-revoke-keys-e2e-mt-616e3ed7-sf646\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.390187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.390168    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"]
Nov 02 01:32:37.487521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.487428    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:37.780231 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:32:37.780201    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760cd3dd_8bdf_471a_a9a1_24518be5643a.slice/crio-00cf3da41699019de33fe56300068dccf6e512b9cfdcdd3405715bebcb7b5b16 WatchSource:0}: Error finding container 00cf3da41699019de33fe56300068dccf6e512b9cfdcdd3405715bebcb7b5b16: Status 404 returned error can't find the container with id 00cf3da41699019de33fe56300068dccf6e512b9cfdcdd3405715bebcb7b5b16
Nov 02 01:32:37.780839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:37.780815    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"]
Nov 02 01:32:38.027225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:38.027145    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"]
Nov 02 01:32:38.032221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:38.032188    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e582780-vp7p8"]
Nov 02 01:32:38.129321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:38.129270    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" event={"ID":"fb40876a-6ba0-40ee-864a-103d087b918e","Type":"ContainerStarted","Data":"54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e"}
Nov 02 01:32:38.129819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:38.129382    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" event={"ID":"fb40876a-6ba0-40ee-864a-103d087b918e","Type":"ContainerStarted","Data":"cf4245ba5461575f8f358469e0a61e4105efb708fe34dc85a4cd169c9f2eb3ee"}
Nov 02 01:32:38.131072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:38.131046    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646" event={"ID":"760cd3dd-8bdf-471a-a9a1-24518be5643a","Type":"ContainerStarted","Data":"00cf3da41699019de33fe56300068dccf6e512b9cfdcdd3405715bebcb7b5b16"}
Nov 02 01:32:38.160771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:38.160726    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" podStartSLOduration=2.160713146 podStartE2EDuration="2.160713146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:38.153984518 +0000 UTC m=+1063.458917358" watchObservedRunningTime="2026-11-02 01:32:38.160713146 +0000 UTC m=+1063.465645972"
Nov 02 01:32:38.407360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:38.402178    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:38.831617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:38.831573    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:38.837833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:38.837807    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:38.848551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:38.848517    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6" path="/var/lib/kubelet/pods/e6ebaa34-a6ca-439e-aaf1-cf3e6564a8e6/volumes"
Nov 02 01:32:39.146562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:39.146475    4851 generic.go:356] "Generic (PLEG): container finished" podID="760cd3dd-8bdf-471a-a9a1-24518be5643a" containerID="f639cfb58d91a4134f2ddb650c6bcaf96fb7ba291716a4a21cc3bc4d343f1969" exitCode=0
Nov 02 01:32:39.146941 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:39.146610    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646" event={"ID":"760cd3dd-8bdf-471a-a9a1-24518be5643a","Type":"ContainerDied","Data":"f639cfb58d91a4134f2ddb650c6bcaf96fb7ba291716a4a21cc3bc4d343f1969"}
Nov 02 01:32:39.147120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:39.147099    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:39.148120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:39.148098    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:32:39.688126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:39.685167    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:40.415707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.415687    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:40.465324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.433144    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:40.595442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.520830    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-xks-ca\") pod \"760cd3dd-8bdf-471a-a9a1-24518be5643a\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") "
Nov 02 01:32:40.595442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.521119    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-openshift-ca\") pod \"760cd3dd-8bdf-471a-a9a1-24518be5643a\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") "
Nov 02 01:32:40.595442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.546376    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/760cd3dd-8bdf-471a-a9a1-24518be5643a-kube-api-access-2h8sj\" (UniqueName: \"kubernetes.io/projected/760cd3dd-8bdf-471a-a9a1-24518be5643a-kube-api-access-2h8sj\") pod \"760cd3dd-8bdf-471a-a9a1-24518be5643a\" (UID: \"760cd3dd-8bdf-471a-a9a1-24518be5643a\") "
Nov 02 01:32:40.595442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.555527    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-openshift-ca" pod "760cd3dd-8bdf-471a-a9a1-24518be5643a" (UID: "760cd3dd-8bdf-471a-a9a1-24518be5643a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:40.685417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.678258    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:40.685417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.678801    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-xks-ca" pod "760cd3dd-8bdf-471a-a9a1-24518be5643a" (UID: "760cd3dd-8bdf-471a-a9a1-24518be5643a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:40.685417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.679065    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:40.691509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.691474    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760cd3dd-8bdf-471a-a9a1-24518be5643a-kube-api-access-2h8sj" pod "760cd3dd-8bdf-471a-a9a1-24518be5643a" (UID: "760cd3dd-8bdf-471a-a9a1-24518be5643a"). InnerVolumeSpecName "kube-api-access-2h8sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:40.779501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.779467    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h8sj\" (UniqueName: \"kubernetes.io/projected/760cd3dd-8bdf-471a-a9a1-24518be5643a-kube-api-access-2h8sj\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:40.779501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.779499    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760cd3dd-8bdf-471a-a9a1-24518be5643a-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:40.810929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:40.810899    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:41.055373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.054347    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"]
Nov 02 01:32:41.065771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.065741    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba78bcec-t2czh"]
Nov 02 01:32:41.169053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.169027    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"
Nov 02 01:32:41.169228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.169057    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646" event={"ID":"760cd3dd-8bdf-471a-a9a1-24518be5643a","Type":"ContainerDied","Data":"00cf3da41699019de33fe56300068dccf6e512b9cfdcdd3405715bebcb7b5b16"}
Nov 02 01:32:41.169228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.169106    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00cf3da41699019de33fe56300068dccf6e512b9cfdcdd3405715bebcb7b5b16"
Nov 02 01:32:41.496202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.496155    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:41.965129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.965089    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:41.995312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.995263    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"]
Nov 02 01:32:41.996076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.996055    4851 cpu_manager.go:414] "containerMap: removing container" podUID="760cd3dd-8bdf-471a-a9a1-24518be5643a" containerName="revoke-keys"
Nov 02 01:32:41.996076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.996076    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="760cd3dd-8bdf-471a-a9a1-24518be5643a" containerName="revoke-keys"
Nov 02 01:32:41.996269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:41.996203    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="760cd3dd-8bdf-471a-a9a1-24518be5643a" containerName="revoke-keys"
Nov 02 01:32:42.000055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.000038    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:32:42.009190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.009158    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2w5\" (UniqueName: \"kubernetes.io/projected/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-kube-api-access-cz2w5\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k\" (UID: \"6d53f56c-35b1-4fcf-be30-b503e0f45a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:32:42.009362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.009332    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k\" (UID: \"6d53f56c-35b1-4fcf-be30-b503e0f45a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:32:42.015996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.015966    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"]
Nov 02 01:32:42.073543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.073512    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-50537f4f\""
Nov 02 01:32:42.094776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.094742    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-50537f4f-dockercfg-8xhfc\""
Nov 02 01:32:42.105170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.105142    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:42.110695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.110669    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2w5\" (UniqueName: \"kubernetes.io/projected/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-kube-api-access-cz2w5\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k\" (UID: \"6d53f56c-35b1-4fcf-be30-b503e0f45a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:32:42.110808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.110745    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k\" (UID: \"6d53f56c-35b1-4fcf-be30-b503e0f45a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:32:42.111466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.111445    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k\" (UID: \"6d53f56c-35b1-4fcf-be30-b503e0f45a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:32:42.122920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.122895    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2w5\" (UniqueName: \"kubernetes.io/projected/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-kube-api-access-cz2w5\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k\" (UID: \"6d53f56c-35b1-4fcf-be30-b503e0f45a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:32:42.324662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.324562    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:32:42.328333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.327782    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:42.478792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.478720    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:42.879513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:42.873183    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cc8105-e17e-4d74-979e-fb0913245396" path="/var/lib/kubelet/pods/51cc8105-e17e-4d74-979e-fb0913245396/volumes"
Nov 02 01:32:43.124628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:43.124051    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"]
Nov 02 01:32:43.290635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:43.290600    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:43.433316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:43.431074    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k" event={"ID":"6d53f56c-35b1-4fcf-be30-b503e0f45a8f","Type":"ContainerStarted","Data":"e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc"}
Nov 02 01:32:43.433316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:43.431148    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k" event={"ID":"6d53f56c-35b1-4fcf-be30-b503e0f45a8f","Type":"ContainerStarted","Data":"d19bc4c46ac0a48d17760487e70f4e235df26ce1ea121966ed4e4cea8b8c4797"}
Nov 02 01:32:43.433316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:43.432439    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:32:43.482316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:43.471707    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k" podStartSLOduration=2.471686447 podStartE2EDuration="2.471686447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:43.468599081 +0000 UTC m=+1068.773532098" watchObservedRunningTime="2026-11-02 01:32:43.471686447 +0000 UTC m=+1068.776619274"
Nov 02 01:32:43.709033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:43.708951    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:43.858355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:43.858128    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:44.278255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:44.278208    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:44.864728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:44.864696    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:44.991935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:44.991905    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:45.133276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:45.133176    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:45.884028 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:45.883995    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:46.576813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.576728    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:46.725323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.725214    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:46.730247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.727763    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"]
Nov 02 01:32:46.751531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.748468    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:46.751531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.750423    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"]
Nov 02 01:32:46.753355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.753333    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:32:46.758791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.757004    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:32:46.758791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.757113    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:32:46.758791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.758458    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:32:46.774986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.774949    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwxqm\" (UniqueName: \"kubernetes.io/projected/d2ea69e3-38be-469c-b453-eab809497a72-kube-api-access-mwxqm\") pod \"maas-api-revoke-sub-b75fc18b-v8t88\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:46.775139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.775052    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75fc18b-v8t88\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:46.775471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.775452    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75fc18b-v8t88\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:46.877059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.876983    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75fc18b-v8t88\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:46.877363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.877334    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75fc18b-v8t88\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:46.877537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.877507    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwxqm\" (UniqueName: \"kubernetes.io/projected/d2ea69e3-38be-469c-b453-eab809497a72-kube-api-access-mwxqm\") pod \"maas-api-revoke-sub-b75fc18b-v8t88\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:46.877987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.877967    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75fc18b-v8t88\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:46.879924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.879899    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75fc18b-v8t88\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:46.890026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:46.890000    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwxqm\" (UniqueName: \"kubernetes.io/projected/d2ea69e3-38be-469c-b453-eab809497a72-kube-api-access-mwxqm\") pod \"maas-api-revoke-sub-b75fc18b-v8t88\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:47.067238 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:47.067206    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:47.304970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:47.304939    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"]
Nov 02 01:32:47.369837 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:32:47.369801    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ea69e3_38be_469c_b453_eab809497a72.slice/crio-5d184414546f2b8fd9d828c98f8b747aedfa33c8919b8fdc87fd1545003b13ba WatchSource:0}: Error finding container 5d184414546f2b8fd9d828c98f8b747aedfa33c8919b8fdc87fd1545003b13ba: Status 404 returned error can't find the container with id 5d184414546f2b8fd9d828c98f8b747aedfa33c8919b8fdc87fd1545003b13ba
Nov 02 01:32:47.378334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:47.377807    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"]
Nov 02 01:32:47.401275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:47.401213    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:47.507242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:47.507201    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88" event={"ID":"d2ea69e3-38be-469c-b453-eab809497a72","Type":"ContainerStarted","Data":"5d184414546f2b8fd9d828c98f8b747aedfa33c8919b8fdc87fd1545003b13ba"}
Nov 02 01:32:47.536189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:47.536155    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:47.833415 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:32:47.821578    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:32:48.026411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:48.026378    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"]
Nov 02 01:32:48.031947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:48.031919    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d40d735-bskfl"]
Nov 02 01:32:48.207802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:48.207650    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:48.521344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:48.521210    4851 generic.go:356] "Generic (PLEG): container finished" podID="d2ea69e3-38be-469c-b453-eab809497a72" containerID="3e4005a4d2c8f1e7f280b68df3bf74ef068fb949efc774ed5d2cf94464ffd59f" exitCode=0
Nov 02 01:32:48.521519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:48.521348    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88" event={"ID":"d2ea69e3-38be-469c-b453-eab809497a72","Type":"ContainerDied","Data":"3e4005a4d2c8f1e7f280b68df3bf74ef068fb949efc774ed5d2cf94464ffd59f"}
Nov 02 01:32:48.864779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:48.864688    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9a1c20-8bb9-46b3-a805-db5cc2b0c44b" path="/var/lib/kubelet/pods/de9a1c20-8bb9-46b3-a805-db5cc2b0c44b/volumes"
Nov 02 01:32:48.904480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:48.904375    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:49.068378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:49.068336    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:49.745414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:49.745096    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:49.937319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:49.930180    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:50.182667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.182630    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:50.429239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.428034    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:50.449938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.449390    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"]
Nov 02 01:32:50.480390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.478604    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-616e3ed7-sf646"]
Nov 02 01:32:50.599946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.599368    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:50.609524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.609249    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88" event={"ID":"d2ea69e3-38be-469c-b453-eab809497a72","Type":"ContainerDied","Data":"5d184414546f2b8fd9d828c98f8b747aedfa33c8919b8fdc87fd1545003b13ba"}
Nov 02 01:32:50.609524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.609328    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d184414546f2b8fd9d828c98f8b747aedfa33c8919b8fdc87fd1545003b13ba"
Nov 02 01:32:50.609524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.609399    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"
Nov 02 01:32:50.648145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.648067    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:50.650752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.650722    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-openshift-ca\") pod \"d2ea69e3-38be-469c-b453-eab809497a72\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") "
Nov 02 01:32:50.650900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.650765    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-xks-ca\") pod \"d2ea69e3-38be-469c-b453-eab809497a72\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") "
Nov 02 01:32:50.651102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.651083    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ea69e3-38be-469c-b453-eab809497a72-kube-api-access-mwxqm\" (UniqueName: \"kubernetes.io/projected/d2ea69e3-38be-469c-b453-eab809497a72-kube-api-access-mwxqm\") pod \"d2ea69e3-38be-469c-b453-eab809497a72\" (UID: \"d2ea69e3-38be-469c-b453-eab809497a72\") "
Nov 02 01:32:50.651102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.651091    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-openshift-ca" pod "d2ea69e3-38be-469c-b453-eab809497a72" (UID: "d2ea69e3-38be-469c-b453-eab809497a72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:50.651979 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.651953    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:50.653797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.653619    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ea69e3-38be-469c-b453-eab809497a72-kube-api-access-mwxqm" pod "d2ea69e3-38be-469c-b453-eab809497a72" (UID: "d2ea69e3-38be-469c-b453-eab809497a72"). InnerVolumeSpecName "kube-api-access-mwxqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:50.654698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.654677    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-xks-ca" pod "d2ea69e3-38be-469c-b453-eab809497a72" (UID: "d2ea69e3-38be-469c-b453-eab809497a72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:50.753928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.753896    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwxqm\" (UniqueName: \"kubernetes.io/projected/d2ea69e3-38be-469c-b453-eab809497a72-kube-api-access-mwxqm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:50.753928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.753927    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ea69e3-38be-469c-b453-eab809497a72-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:50.850050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:50.850015    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="760cd3dd-8bdf-471a-a9a1-24518be5643a" path="/var/lib/kubelet/pods/760cd3dd-8bdf-471a-a9a1-24518be5643a/volumes"
Nov 02 01:32:51.124239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:51.124204    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:51.906320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:51.892019    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:52.305997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.305921    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4" podUID="cd8f7c8f-8941-4921-8103-82a2414b77f0" containerName="payload-pre-processing" containerID="cri-o://3f17d507c20f5ea6eb52ffadad28ee45c03ba8143f9cddf3dc3d3ea052616425" gracePeriod=25
Nov 02 01:32:52.599315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.589818    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:52.742762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.742618    4851 generic.go:356] "Generic (PLEG): container finished" podID="20b253b7-6535-426d-814c-9c509d7c2992" containerID="40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36" exitCode=0
Nov 02 01:32:52.742762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.742685    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" event={"ID":"20b253b7-6535-426d-814c-9c509d7c2992","Type":"ContainerDied","Data":"40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36"}
Nov 02 01:32:52.743494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.743222    4851 scope.go:122] "RemoveContainer" containerID="40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36"
Nov 02 01:32:52.745836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.745046    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:32:52.745836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.745113    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:32:52.745836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.745145    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:32:52.748504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.748382    4851 generic.go:356] "Generic (PLEG): container finished" podID="cd8f7c8f-8941-4921-8103-82a2414b77f0" containerID="3f17d507c20f5ea6eb52ffadad28ee45c03ba8143f9cddf3dc3d3ea052616425" exitCode=0
Nov 02 01:32:52.748623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.748410    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4" event={"ID":"cd8f7c8f-8941-4921-8103-82a2414b77f0","Type":"ContainerDied","Data":"3f17d507c20f5ea6eb52ffadad28ee45c03ba8143f9cddf3dc3d3ea052616425"}
Nov 02 01:32:52.784577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.784549    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:32:52.826650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.825270    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8f7c8f-8941-4921-8103-82a2414b77f0-kube-api-access-9mj4b\" (UniqueName: \"kubernetes.io/projected/cd8f7c8f-8941-4921-8103-82a2414b77f0-kube-api-access-9mj4b\") pod \"cd8f7c8f-8941-4921-8103-82a2414b77f0\" (UID: \"cd8f7c8f-8941-4921-8103-82a2414b77f0\") "
Nov 02 01:32:52.826650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.825471    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd8f7c8f-8941-4921-8103-82a2414b77f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f7c8f-8941-4921-8103-82a2414b77f0-plugins-config-volume\") pod \"cd8f7c8f-8941-4921-8103-82a2414b77f0\" (UID: \"cd8f7c8f-8941-4921-8103-82a2414b77f0\") "
Nov 02 01:32:52.826650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.825885    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8f7c8f-8941-4921-8103-82a2414b77f0-plugins-config-volume" pod "cd8f7c8f-8941-4921-8103-82a2414b77f0" (UID: "cd8f7c8f-8941-4921-8103-82a2414b77f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:52.826650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.826581    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd8f7c8f-8941-4921-8103-82a2414b77f0-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:52.828921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.828893    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8f7c8f-8941-4921-8103-82a2414b77f0-kube-api-access-9mj4b" pod "cd8f7c8f-8941-4921-8103-82a2414b77f0" (UID: "cd8f7c8f-8941-4921-8103-82a2414b77f0"). InnerVolumeSpecName "kube-api-access-9mj4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:52.929075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:52.928846    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mj4b\" (UniqueName: \"kubernetes.io/projected/cd8f7c8f-8941-4921-8103-82a2414b77f0-kube-api-access-9mj4b\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:32:53.290900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:53.290866    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:53.766590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:53.766555    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" event={"ID":"20b253b7-6535-426d-814c-9c509d7c2992","Type":"ContainerStarted","Data":"65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e"}
Nov 02 01:32:53.768114 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:53.768094    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"
Nov 02 01:32:53.768251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:53.768152    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4" event={"ID":"cd8f7c8f-8941-4921-8103-82a2414b77f0","Type":"ContainerDied","Data":"4ffabdd30096b5708adf64630547dc778e7233406512fabfb947239081d5b7f3"}
Nov 02 01:32:53.768251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:53.768208    4851 scope.go:122] "RemoveContainer" containerID="3f17d507c20f5ea6eb52ffadad28ee45c03ba8143f9cddf3dc3d3ea052616425"
Nov 02 01:32:53.810175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:53.810135    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"]
Nov 02 01:32:53.817902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:53.817874    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-616e3ed7-56bf47c4f5-ng2l4"]
Nov 02 01:32:54.021070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:54.020984    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:54.430367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:54.430338    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:32:54.431370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:54.431345    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:32:54.431497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:54.431424    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 01:32:54.759366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:54.750886    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:54.942327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:54.937440    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd8f7c8f-8941-4921-8103-82a2414b77f0" path="/var/lib/kubelet/pods/cd8f7c8f-8941-4921-8103-82a2414b77f0/volumes"
Nov 02 01:32:55.431052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:55.431013    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:32:55.431525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:55.431071    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 01:32:55.486202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:55.486176    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:32:55.726390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:55.711712    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:56.026264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:56.026170    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"]
Nov 02 01:32:56.031503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:56.031481    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93fdcd4a-5hvms"]
Nov 02 01:32:56.431182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:56.431152    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:32:56.431631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:56.431211    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 01:32:56.451154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:56.451117    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:56.462435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:56.462404    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:32:56.869122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:56.869039    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a763458-101d-468e-bfd3-c82efd747be1" path="/var/lib/kubelet/pods/4a763458-101d-468e-bfd3-c82efd747be1/volumes"
Nov 02 01:32:57.059469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:57.059244    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:57.223315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:57.214113    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:57.431061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:57.431026    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:32:57.431307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:57.431109    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 01:32:57.527741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:57.527617    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:57.665231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:57.665191    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:57.908126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:57.907785    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:58.095600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:58.095561    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:58.347638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:58.347602    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"]
Nov 02 01:32:58.348250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:58.348216    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy" containerID="cri-o://6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3" gracePeriod=30
Nov 02 01:32:58.463321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:58.460471    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:58.463321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:58.462886    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:32:58.463321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:58.462941    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 01:32:58.690663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:58.690624    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:58.821271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:58.821225    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:59.271586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:59.271550    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:32:59.441756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:59.430379    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 02 01:32:59.441756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:32:59.430451    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 02 01:33:00.024128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.024088    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:00.435277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.435252    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:33:00.436448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.436429    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:33:00.681518 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.681480    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:00.784410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.782329    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"]
Nov 02 01:33:00.784410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.783595    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d2ea69e3-38be-469c-b453-eab809497a72" containerName="revoke-keys"
Nov 02 01:33:00.784410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.783620    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ea69e3-38be-469c-b453-eab809497a72" containerName="revoke-keys"
Nov 02 01:33:00.784410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.783671    4851 cpu_manager.go:414] "containerMap: removing container" podUID="cd8f7c8f-8941-4921-8103-82a2414b77f0" containerName="payload-pre-processing"
Nov 02 01:33:00.784410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.783680    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8f7c8f-8941-4921-8103-82a2414b77f0" containerName="payload-pre-processing"
Nov 02 01:33:00.784410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.783883    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ea69e3-38be-469c-b453-eab809497a72" containerName="revoke-keys"
Nov 02 01:33:00.784410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.783902    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8f7c8f-8941-4921-8103-82a2414b77f0" containerName="payload-pre-processing"
Nov 02 01:33:00.805129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.805098    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:00.833164 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.807701    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"]
Nov 02 01:33:00.847121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.847074    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:33:00.847418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.847401    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:33:00.847642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.847627    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:33:00.847794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.847782    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:33:00.855238 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.855192    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:00.985343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.985278    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c918649-6jq6p\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:00.985515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.985434    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c918649-6jq6p\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:00.985515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:00.985480    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7p68\" (UniqueName: \"kubernetes.io/projected/465ac986-4ebc-4351-b221-51e64500cadb-kube-api-access-h7p68\") pod \"maas-api-revoke-sub-0c918649-6jq6p\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:01.087195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:01.087110    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7p68\" (UniqueName: \"kubernetes.io/projected/465ac986-4ebc-4351-b221-51e64500cadb-kube-api-access-h7p68\") pod \"maas-api-revoke-sub-0c918649-6jq6p\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:01.087594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:01.087385    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c918649-6jq6p\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:01.087594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:01.087437    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c918649-6jq6p\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:01.088046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:01.088021    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c918649-6jq6p\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:01.089983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:01.089954    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c918649-6jq6p\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:01.099199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:01.099176    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7p68\" (UniqueName: \"kubernetes.io/projected/465ac986-4ebc-4351-b221-51e64500cadb-kube-api-access-h7p68\") pod \"maas-api-revoke-sub-0c918649-6jq6p\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:01.189816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:01.189786    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:01.486317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:01.470003    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:01.568734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:01.568697    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"]
Nov 02 01:33:01.585100 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:33:01.584916    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465ac986_4ebc_4351_b221_51e64500cadb.slice/crio-78a4e962eaf793d9eeebb8ee97f0ea3fbe922588d80a9c7a3c0c43ce65d04aad WatchSource:0}: Error finding container 78a4e962eaf793d9eeebb8ee97f0ea3fbe922588d80a9c7a3c0c43ce65d04aad: Status 404 returned error can't find the container with id 78a4e962eaf793d9eeebb8ee97f0ea3fbe922588d80a9c7a3c0c43ce65d04aad
Nov 02 01:33:02.211897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:02.211859    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p" event={"ID":"465ac986-4ebc-4351-b221-51e64500cadb","Type":"ContainerStarted","Data":"ac5d528b4ea64d879ab4efcebb3ede877a995cffe1cf7436acb47a6c832a14a4"}
Nov 02 01:33:02.212328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:02.211952    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p" event={"ID":"465ac986-4ebc-4351-b221-51e64500cadb","Type":"ContainerStarted","Data":"78a4e962eaf793d9eeebb8ee97f0ea3fbe922588d80a9c7a3c0c43ce65d04aad"}
Nov 02 01:33:02.234391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:02.234334    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p" podStartSLOduration=2.23431724 podStartE2EDuration="2.23431724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:02.229531733 +0000 UTC m=+1087.534464573" watchObservedRunningTime="2026-11-02 01:33:02.23431724 +0000 UTC m=+1087.539250065"
Nov 02 01:33:02.257730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:02.257700    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:02.941940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:02.941902    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:03.161277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.161227    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:03.223391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.223272    4851 generic.go:356] "Generic (PLEG): container finished" podID="465ac986-4ebc-4351-b221-51e64500cadb" containerID="ac5d528b4ea64d879ab4efcebb3ede877a995cffe1cf7436acb47a6c832a14a4" exitCode=0
Nov 02 01:33:03.223391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.223323    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p" event={"ID":"465ac986-4ebc-4351-b221-51e64500cadb","Type":"ContainerDied","Data":"ac5d528b4ea64d879ab4efcebb3ede877a995cffe1cf7436acb47a6c832a14a4"}
Nov 02 01:33:03.485795 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.485691    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:03.741677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.741616    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:33:03.760030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760006    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-certs\") pod \"83e32e09-9992-4e33-906c-cb088eb20ddc\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") "
Nov 02 01:33:03.760146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760048    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83e32e09-9992-4e33-906c-cb088eb20ddc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83e32e09-9992-4e33-906c-cb088eb20ddc-istiod-ca-cert\") pod \"83e32e09-9992-4e33-906c-cb088eb20ddc\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") "
Nov 02 01:33:03.760146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760069    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-token\" (UniqueName: \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-token\") pod \"83e32e09-9992-4e33-906c-cb088eb20ddc\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") "
Nov 02 01:33:03.760231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760185    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-socket\") pod \"83e32e09-9992-4e33-906c-cb088eb20ddc\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") "
Nov 02 01:33:03.760398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760380    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-kube-api-access-brl9f\" (UniqueName: \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-kube-api-access-brl9f\") pod \"83e32e09-9992-4e33-906c-cb088eb20ddc\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") "
Nov 02 01:33:03.760513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760378    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-certs" pod "83e32e09-9992-4e33-906c-cb088eb20ddc" (UID: "83e32e09-9992-4e33-906c-cb088eb20ddc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:33:03.760559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760460    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-socket" pod "83e32e09-9992-4e33-906c-cb088eb20ddc" (UID: "83e32e09-9992-4e33-906c-cb088eb20ddc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:33:03.760559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760482    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e32e09-9992-4e33-906c-cb088eb20ddc-istiod-ca-cert" pod "83e32e09-9992-4e33-906c-cb088eb20ddc" (UID: "83e32e09-9992-4e33-906c-cb088eb20ddc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:03.760559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760497    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-podinfo\") pod \"83e32e09-9992-4e33-906c-cb088eb20ddc\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") "
Nov 02 01:33:03.760673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760577    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-envoy\") pod \"83e32e09-9992-4e33-906c-cb088eb20ddc\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") "
Nov 02 01:33:03.760793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760696    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-data\") pod \"83e32e09-9992-4e33-906c-cb088eb20ddc\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") "
Nov 02 01:33:03.760793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760741    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-credential-socket\") pod \"83e32e09-9992-4e33-906c-cb088eb20ddc\" (UID: \"83e32e09-9992-4e33-906c-cb088eb20ddc\") "
Nov 02 01:33:03.761022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.760990    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-data" pod "83e32e09-9992-4e33-906c-cb088eb20ddc" (UID: "83e32e09-9992-4e33-906c-cb088eb20ddc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:33:03.761267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.761243    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-credential-socket" pod "83e32e09-9992-4e33-906c-cb088eb20ddc" (UID: "83e32e09-9992-4e33-906c-cb088eb20ddc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:33:03.761721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.761694    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83e32e09-9992-4e33-906c-cb088eb20ddc-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:03.761815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.761726    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:03.761815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.761807    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:03.761933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.761823    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:03.761933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.761836    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:03.762554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.762529    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-token" pod "83e32e09-9992-4e33-906c-cb088eb20ddc" (UID: "83e32e09-9992-4e33-906c-cb088eb20ddc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:03.762833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.762813    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-podinfo" pod "83e32e09-9992-4e33-906c-cb088eb20ddc" (UID: "83e32e09-9992-4e33-906c-cb088eb20ddc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:33:03.762904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.762868    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-kube-api-access-brl9f" pod "83e32e09-9992-4e33-906c-cb088eb20ddc" (UID: "83e32e09-9992-4e33-906c-cb088eb20ddc"). InnerVolumeSpecName "kube-api-access-brl9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:03.762991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.762975    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-envoy" pod "83e32e09-9992-4e33-906c-cb088eb20ddc" (UID: "83e32e09-9992-4e33-906c-cb088eb20ddc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:33:03.863362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.863330    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:03.863362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.863358    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brl9f\" (UniqueName: \"kubernetes.io/projected/83e32e09-9992-4e33-906c-cb088eb20ddc-kube-api-access-brl9f\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:03.863362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.863367    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:03.863586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.863376    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83e32e09-9992-4e33-906c-cb088eb20ddc-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:03.921135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:03.921105    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:04.235801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.235766    4851 generic.go:356] "Generic (PLEG): container finished" podID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerID="6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3" exitCode=0
Nov 02 01:33:04.236352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.235829    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"
Nov 02 01:33:04.236352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.235865    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" event={"ID":"83e32e09-9992-4e33-906c-cb088eb20ddc","Type":"ContainerDied","Data":"6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3"}
Nov 02 01:33:04.236352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.235925    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw" event={"ID":"83e32e09-9992-4e33-906c-cb088eb20ddc","Type":"ContainerDied","Data":"afff7bfb6a1c65ec470c792b404473b91194b66ae192caa4ddfe315d39ccd3d3"}
Nov 02 01:33:04.236352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.235953    4851 scope.go:122] "RemoveContainer" containerID="6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3"
Nov 02 01:33:04.252940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.252916    4851 scope.go:122] "RemoveContainer" containerID="7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b"
Nov 02 01:33:04.268505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.268483    4851 scope.go:122] "RemoveContainer" containerID="6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3"
Nov 02 01:33:04.268805 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:33:04.268781    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3\": container with ID starting with 6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3 not found: ID does not exist" containerID="6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3"
Nov 02 01:33:04.268922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.268812    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3"} err="failed to get container status \"6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3\": rpc error: code = NotFound desc = could not find container \"6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3\": container with ID starting with 6b1c1a9f2e6767a24e09fac962b0eb19e607ecf851d20eff25c763e328a8b5d3 not found: ID does not exist"
Nov 02 01:33:04.268922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.268831    4851 scope.go:122] "RemoveContainer" containerID="7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b"
Nov 02 01:33:04.269073 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:33:04.269058    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b\": container with ID starting with 7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b not found: ID does not exist" containerID="7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b"
Nov 02 01:33:04.269121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.269076    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b"} err="failed to get container status \"7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b\": rpc error: code = NotFound desc = could not find container \"7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b\": container with ID starting with 7f1736ade0ec54ceb13ebec26550dcea9ef6ee10611eaa5555dabc3c6069b88b not found: ID does not exist"
Nov 02 01:33:04.271381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.271360    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"]
Nov 02 01:33:04.281612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.281586    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-616e3ed7-openshift-default-677bc54677-plhfw"]
Nov 02 01:33:04.401916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.401893    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:04.480435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.480405    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465ac986-4ebc-4351-b221-51e64500cadb-kube-api-access-h7p68\" (UniqueName: \"kubernetes.io/projected/465ac986-4ebc-4351-b221-51e64500cadb-kube-api-access-h7p68\") pod \"465ac986-4ebc-4351-b221-51e64500cadb\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") "
Nov 02 01:33:04.480625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.480551    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-xks-ca\") pod \"465ac986-4ebc-4351-b221-51e64500cadb\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") "
Nov 02 01:33:04.480776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.480755    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-openshift-ca\") pod \"465ac986-4ebc-4351-b221-51e64500cadb\" (UID: \"465ac986-4ebc-4351-b221-51e64500cadb\") "
Nov 02 01:33:04.481118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.481094    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-openshift-ca" pod "465ac986-4ebc-4351-b221-51e64500cadb" (UID: "465ac986-4ebc-4351-b221-51e64500cadb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:04.481590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.481564    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:04.482896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.482876    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-xks-ca" pod "465ac986-4ebc-4351-b221-51e64500cadb" (UID: "465ac986-4ebc-4351-b221-51e64500cadb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:04.483257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.483242    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465ac986-4ebc-4351-b221-51e64500cadb-kube-api-access-h7p68" pod "465ac986-4ebc-4351-b221-51e64500cadb" (UID: "465ac986-4ebc-4351-b221-51e64500cadb"). InnerVolumeSpecName "kube-api-access-h7p68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:04.603413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.602739    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7p68\" (UniqueName: \"kubernetes.io/projected/465ac986-4ebc-4351-b221-51e64500cadb-kube-api-access-h7p68\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:04.603413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.602781    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465ac986-4ebc-4351-b221-51e64500cadb-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:04.715269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.715001    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:04.883213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:04.883133    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" path="/var/lib/kubelet/pods/83e32e09-9992-4e33-906c-cb088eb20ddc/volumes"
Nov 02 01:33:05.246940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:05.246907    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"
Nov 02 01:33:05.246940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:05.246926    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p" event={"ID":"465ac986-4ebc-4351-b221-51e64500cadb","Type":"ContainerDied","Data":"78a4e962eaf793d9eeebb8ee97f0ea3fbe922588d80a9c7a3c0c43ce65d04aad"}
Nov 02 01:33:05.247498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:05.246985    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78a4e962eaf793d9eeebb8ee97f0ea3fbe922588d80a9c7a3c0c43ce65d04aad"
Nov 02 01:33:05.425146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:05.425114    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:05.659087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:05.658953    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:05.976144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:05.976106    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:06.229506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.229400    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:06.357207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.357173    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"]
Nov 02 01:33:06.357866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.357850    4851 cpu_manager.go:414] "containerMap: removing container" podUID="465ac986-4ebc-4351-b221-51e64500cadb" containerName="revoke-keys"
Nov 02 01:33:06.357921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.357866    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465ac986-4ebc-4351-b221-51e64500cadb" containerName="revoke-keys"
Nov 02 01:33:06.357921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.357881    4851 cpu_manager.go:414] "containerMap: removing container" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy"
Nov 02 01:33:06.357921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.357886    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy"
Nov 02 01:33:06.358040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.358031    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy"
Nov 02 01:33:06.358073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.358043    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="465ac986-4ebc-4351-b221-51e64500cadb" containerName="revoke-keys"
Nov 02 01:33:06.358073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.358050    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy"
Nov 02 01:33:06.362226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.362208    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.379075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.379052    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"]
Nov 02 01:33:06.393596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.393564    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:06.435027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.434988    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-certs\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.435185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.435151    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-socket\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.435258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.435236    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-data\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.435431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.435408    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-token\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.435494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.435482    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnb4\" (UniqueName: \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-kube-api-access-spnb4\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.435556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.435539    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-podinfo\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.435674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.435655    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-envoy\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.435733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.435721    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-credential-socket\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.435811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.435797    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istiod-ca-cert\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.440557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.440535    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2ebe99-openshift-default-dockercfg-p974f\""
Nov 02 01:33:06.564365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.561685    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:06.564365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.563267    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istiod-ca-cert\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.564365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.563348    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-certs\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.564365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.563504    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-socket\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.564365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.563604    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-data\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.564365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.563741    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-token\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.564365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.563823    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnb4\" (UniqueName: \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-kube-api-access-spnb4\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.564365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.563872    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-podinfo\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.564365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.563987    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-envoy\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.564365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.564103    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-credential-socket\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.576325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.573764    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-data\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.576325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.574435    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istiod-ca-cert\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.599315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.595622    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-certs\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.599551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.596620    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-socket\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.603367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.597280    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-credential-socket\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.622985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.622902    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-envoy\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.641350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.623672    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-podinfo\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.688313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.684889    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-token\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.688313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.685489    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnb4\" (UniqueName: \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-kube-api-access-spnb4\") pod \"e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") " pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:06.688313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:06.686586    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:07.154366 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:33:07.154331    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7f563e_5da1_4854_a8fa_e3931a462b3e.slice/crio-17e654075638d73bc06681928d54d54557b8814b41abb47d4ae72b3d89770abf WatchSource:0}: Error finding container 17e654075638d73bc06681928d54d54557b8814b41abb47d4ae72b3d89770abf: Status 404 returned error can't find the container with id 17e654075638d73bc06681928d54d54557b8814b41abb47d4ae72b3d89770abf
Nov 02 01:33:07.156435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:07.156415    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"]
Nov 02 01:33:07.157411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:07.157372    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:07.157511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:07.157455    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:07.157511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:07.157497    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:07.213664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:07.213634    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:07.322596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:07.322499    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" event={"ID":"9b7f563e-5da1-4854-a8fa-e3931a462b3e","Type":"ContainerStarted","Data":"17e654075638d73bc06681928d54d54557b8814b41abb47d4ae72b3d89770abf"}
Nov 02 01:33:08.009626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:08.009591    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:08.334117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:08.334027    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" event={"ID":"9b7f563e-5da1-4854-a8fa-e3931a462b3e","Type":"ContainerStarted","Data":"e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029"}
Nov 02 01:33:08.361380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:08.361318    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" podStartSLOduration=2.361282749 podStartE2EDuration="2.361282749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:08.358659163 +0000 UTC m=+1093.663591990" watchObservedRunningTime="2026-11-02 01:33:08.361282749 +0000 UTC m=+1093.666215872"
Nov 02 01:33:08.689933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:08.689899    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:08.696625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:08.696593    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:08.812220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:08.812188    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:09.361995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:09.353865    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:09.361995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:09.357785    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:33:09.795585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:09.795551    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:10.720159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:10.720124    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:10.858198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:10.858148    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:11.323540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:11.323497    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:11.508362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:11.501936    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:12.030454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:12.030361    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:12.245217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:12.215967    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:13.181674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.175854    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:13.289352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.272762    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:13.290711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.290661    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:13.299001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.298943    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:13.372017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.371963    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:13.433372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.432841    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:33:13.433372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.433075    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:13.443518 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.442542    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:33:13.475334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.472210    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:13.475627 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.475586    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:13.590349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.588722    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"]
Nov 02 01:33:13.713366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.712527    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:13.740318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.722526    4851 cpu_manager.go:414] "containerMap: removing container" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy"
Nov 02 01:33:13.740318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.722555    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e32e09-9992-4e33-906c-cb088eb20ddc" containerName="istio-proxy"
Nov 02 01:33:13.833322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.818240    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:13.833579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.833549    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:33:13.873519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.873485    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2ebe99\""
Nov 02 01:33:13.873978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.873956    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2ebe99-dockercfg-mfzld\""
Nov 02 01:33:13.885409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.885373    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"]
Nov 02 01:33:13.955501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.955458    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:13.956492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.956463    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e5798a-381e-4029-9b0a-364e77864ee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb\" (UID: \"84e5798a-381e-4029-9b0a-364e77864ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:33:13.956929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:13.956910    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82b2k\" (UniqueName: \"kubernetes.io/projected/84e5798a-381e-4029-9b0a-364e77864ee5-kube-api-access-82b2k\") pod \"payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb\" (UID: \"84e5798a-381e-4029-9b0a-364e77864ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:33:14.078342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.057885    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded"
Nov 02 01:33:14.078544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.078378    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e5798a-381e-4029-9b0a-364e77864ee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb\" (UID: \"84e5798a-381e-4029-9b0a-364e77864ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:33:14.078544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.078460    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82b2k\" (UniqueName: \"kubernetes.io/projected/84e5798a-381e-4029-9b0a-364e77864ee5-kube-api-access-82b2k\") pod \"payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb\" (UID: \"84e5798a-381e-4029-9b0a-364e77864ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:33:14.087417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.079736    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e5798a-381e-4029-9b0a-364e77864ee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb\" (UID: \"84e5798a-381e-4029-9b0a-364e77864ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:33:14.108225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.108187    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:14.138315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.134560    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"]
Nov 02 01:33:14.203755 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.203718    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:14.241432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.213701    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:33:14.241432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.214457    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:33:14.241432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.214565    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:33:14.241432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.214690    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:33:14.268826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.248458    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:33:14.330990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.314508    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:33:14.330990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.317564    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:33:14.412388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.370108    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:33:14.412388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.373115    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:14.412388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.381572    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:33:14.479324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.475319    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:14.479324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.475727    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:33:14.479324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.475827    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:33:14.479324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.476141    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:33:14.479324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.476905    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:33:14.479324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.476967    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:33:14.479324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.477005    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:33:14.479324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.477043    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:33:14.534732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.514237    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4adfa880-dvpc2\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:14.534732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.531775    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4adfa880-dvpc2\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:14.534732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.531974    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2pfr\" (UniqueName: \"kubernetes.io/projected/825c1938-b9c8-4297-8ad8-d4ebe83ad844-kube-api-access-l2pfr\") pod \"maas-api-revoke-sub-4adfa880-dvpc2\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:14.657436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.569959    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:33:14.657436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.611807    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:33:14.657436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.624317    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"]
Nov 02 01:33:14.726351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.701998    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4adfa880-dvpc2\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:14.726351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.702191    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2pfr\" (UniqueName: \"kubernetes.io/projected/825c1938-b9c8-4297-8ad8-d4ebe83ad844-kube-api-access-l2pfr\") pod \"maas-api-revoke-sub-4adfa880-dvpc2\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:14.747347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.730026    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4adfa880-dvpc2\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:14.770317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.749498    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:33:14.779043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.778957    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:33:14.779723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.771161    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:14.823057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.795121    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4adfa880-dvpc2\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:14.994544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.818673    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:33:14.999420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.945520    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82b2k\" (UniqueName: \"kubernetes.io/projected/84e5798a-381e-4029-9b0a-364e77864ee5-kube-api-access-82b2k\") pod \"payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb\" (UID: \"84e5798a-381e-4029-9b0a-364e77864ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:33:15.044383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.950401    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2pfr\" (UniqueName: \"kubernetes.io/projected/825c1938-b9c8-4297-8ad8-d4ebe83ad844-kube-api-access-l2pfr\") pod \"maas-api-revoke-sub-4adfa880-dvpc2\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:15.044383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:14.959369    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:15.071902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.071872    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:33:15.134324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.133822    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:33:15.136129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.135704    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:33:15.141141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.140701    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:33:15.146768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.144118    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:33:15.191189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.191158    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:33:15.248905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.248818    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:15.272237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.272191    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:15.279599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.279563    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:33:15.286377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.281766    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4adfa880-dvpc2\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:15.286377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.282932    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:33:15.323447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.317826    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:15.386586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.386545    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:33:15.415084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.412246    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:33:15.420494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.420471    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:33:15.435730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.435699    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:33:15.436568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.436550    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:33:15.441314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.436786    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:33:15.482947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.482917    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:33:15.487065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.487036    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:15.562951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.562693    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:15.956542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:15.955906    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"]
Nov 02 01:33:16.100344 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:33:16.100277    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825c1938_b9c8_4297_8ad8_d4ebe83ad844.slice/crio-41fa44925d0edf6618d7dacad73f1944fb50f4f56db16791c4411075364b69a7 WatchSource:0}: Error finding container 41fa44925d0edf6618d7dacad73f1944fb50f4f56db16791c4411075364b69a7: Status 404 returned error can't find the container with id 41fa44925d0edf6618d7dacad73f1944fb50f4f56db16791c4411075364b69a7
Nov 02 01:33:16.101121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:16.101097    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"]
Nov 02 01:33:16.519617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:16.519575    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:16.705987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:16.705901    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2" event={"ID":"825c1938-b9c8-4297-8ad8-d4ebe83ad844","Type":"ContainerStarted","Data":"41fa44925d0edf6618d7dacad73f1944fb50f4f56db16791c4411075364b69a7"}
Nov 02 01:33:16.708112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:16.708048    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb" event={"ID":"84e5798a-381e-4029-9b0a-364e77864ee5","Type":"ContainerStarted","Data":"6aaaac4332ebced9b215f5ca88b481e06d3a8258394c1225c9bcf758d01292e1"}
Nov 02 01:33:16.708112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:16.708106    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb" event={"ID":"84e5798a-381e-4029-9b0a-364e77864ee5","Type":"ContainerStarted","Data":"e6dec0fb4c644e8d3e7c69e040a2c7c249b6579f711e7df0b0334713f4276ed0"}
Nov 02 01:33:16.709514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:16.709395    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:33:16.730905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:16.730863    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb" podStartSLOduration=4.730849255 podStartE2EDuration="4.730849255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:16.727167754 +0000 UTC m=+1102.032100582" watchObservedRunningTime="2026-11-02 01:33:16.730849255 +0000 UTC m=+1102.035782082"
Nov 02 01:33:17.324252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:17.324209    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:17.727544 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:33:17.727460    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:33:17.734022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:17.733987    4851 generic.go:356] "Generic (PLEG): container finished" podID="825c1938-b9c8-4297-8ad8-d4ebe83ad844" containerID="0e708bdd297115b71eadf4592f08cf159a43d34ec9136198113c45cf4291572d" exitCode=0
Nov 02 01:33:17.734174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:17.734061    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2" event={"ID":"825c1938-b9c8-4297-8ad8-d4ebe83ad844","Type":"ContainerDied","Data":"0e708bdd297115b71eadf4592f08cf159a43d34ec9136198113c45cf4291572d"}
Nov 02 01:33:18.037530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:18.037448    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:18.449809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:18.449767    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:18.735620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:18.735538    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:19.058939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.058913    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:19.204134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.204099    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825c1938-b9c8-4297-8ad8-d4ebe83ad844-kube-api-access-l2pfr\" (UniqueName: \"kubernetes.io/projected/825c1938-b9c8-4297-8ad8-d4ebe83ad844-kube-api-access-l2pfr\") pod \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") "
Nov 02 01:33:19.204436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.204416    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-xks-ca\") pod \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") "
Nov 02 01:33:19.204606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.204589    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-openshift-ca\") pod \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\" (UID: \"825c1938-b9c8-4297-8ad8-d4ebe83ad844\") "
Nov 02 01:33:19.205185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.205145    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-openshift-ca" pod "825c1938-b9c8-4297-8ad8-d4ebe83ad844" (UID: "825c1938-b9c8-4297-8ad8-d4ebe83ad844"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:19.215313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.206051    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:19.215313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.211172    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825c1938-b9c8-4297-8ad8-d4ebe83ad844-kube-api-access-l2pfr" pod "825c1938-b9c8-4297-8ad8-d4ebe83ad844" (UID: "825c1938-b9c8-4297-8ad8-d4ebe83ad844"). InnerVolumeSpecName "kube-api-access-l2pfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:19.215313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.214795    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-xks-ca" pod "825c1938-b9c8-4297-8ad8-d4ebe83ad844" (UID: "825c1938-b9c8-4297-8ad8-d4ebe83ad844"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:19.308339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.308229    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825c1938-b9c8-4297-8ad8-d4ebe83ad844-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:19.308339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.308306    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2pfr\" (UniqueName: \"kubernetes.io/projected/825c1938-b9c8-4297-8ad8-d4ebe83ad844-kube-api-access-l2pfr\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:33:19.587661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.587576    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:19.810922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.810883    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2" event={"ID":"825c1938-b9c8-4297-8ad8-d4ebe83ad844","Type":"ContainerDied","Data":"41fa44925d0edf6618d7dacad73f1944fb50f4f56db16791c4411075364b69a7"}
Nov 02 01:33:19.810922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.810914    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"
Nov 02 01:33:19.811413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:19.810978    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41fa44925d0edf6618d7dacad73f1944fb50f4f56db16791c4411075364b69a7"
Nov 02 01:33:20.424645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:20.424604    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:21.238813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:21.238779    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:21.382153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:21.382110    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:21.608196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:21.608100    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:22.227583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:22.227552    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:23.024675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:23.024646    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:23.717080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:23.717049    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:23.918998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:23.918963    4851 generic.go:356] "Generic (PLEG): container finished" podID="fb40876a-6ba0-40ee-864a-103d087b918e" containerID="54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e" exitCode=0
Nov 02 01:33:23.919179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:23.919036    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" event={"ID":"fb40876a-6ba0-40ee-864a-103d087b918e","Type":"ContainerDied","Data":"54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e"}
Nov 02 01:33:23.919547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:23.919526    4851 scope.go:122] "RemoveContainer" containerID="54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e"
Nov 02 01:33:23.921285 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:23.921259    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:23.921381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:23.921350    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:23.921419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:23.921384    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:24.026261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:24.026215    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:24.151328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:24.149375    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:33:24.951315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:24.942342    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" event={"ID":"fb40876a-6ba0-40ee-864a-103d087b918e","Type":"ContainerStarted","Data":"ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12"}
Nov 02 01:33:25.057313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:25.056899    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:25.832042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:25.832007    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:33:25.839317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:25.833429    4851 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:33:25.839317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:25.833499    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 01:33:26.016644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:26.016608    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:26.084854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:26.084768    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"]
Nov 02 01:33:26.175843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:26.175804    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c040c3d5-qqpwz"]
Nov 02 01:33:26.832163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:26.832125    4851 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:33:26.832429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:26.832193    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 01:33:26.867323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:26.867271    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb" path="/var/lib/kubelet/pods/c47d4cb3-e2b8-40cb-8ba6-73c8bc1baacb/volumes"
Nov 02 01:33:27.144986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:27.144900    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:27.837781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:27.837741    4851 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:33:27.838132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:27.838097    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 01:33:28.803432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:28.803405    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:33:28.827343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:28.827307    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:28.832743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:28.832596    4851 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:33:28.833052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:28.833016    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 01:33:29.835314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:29.834724    4851 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:33:29.835314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:29.834810    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 01:33:30.315357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:30.250459    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:30.415314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:30.411156    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:30.859456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:30.859416    4851 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 01:33:30.860104 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:30.860068    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 01:33:30.904581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:30.904542    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:31.140618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.140524    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:31.261909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.261866    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:31.304489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.304442    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:31.374318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.373830    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:31.432386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.418388    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:31.432386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.426870    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:31.446416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.445379    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:31.468062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.467575    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:33:31.468862 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.468818    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:31.709947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.708928    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:31.882314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.882098    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7f78h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:33:31.882314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:31.882173    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:32.215151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:32.215117    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:32.528036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:32.527951    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:33:32.528838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:32.528815    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:33:32.529717 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:32.529693    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:33:33.125951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:33.125916    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:33.815453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:33.815423    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:34.748880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:34.748847    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:35.561008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:35.560967    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:35.945759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:35.945727    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:36.314075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:36.313996    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:36.628673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:36.628585    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:37.026173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.026136    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:37.046738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.046698    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"]
Nov 02 01:33:37.047531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.047513    4851 cpu_manager.go:414] "containerMap: removing container" podUID="825c1938-b9c8-4297-8ad8-d4ebe83ad844" containerName="revoke-keys"
Nov 02 01:33:37.047531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.047530    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825c1938-b9c8-4297-8ad8-d4ebe83ad844" containerName="revoke-keys"
Nov 02 01:33:37.047698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.047688    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="825c1938-b9c8-4297-8ad8-d4ebe83ad844" containerName="revoke-keys"
Nov 02 01:33:37.052725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.052707    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.068152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.068126    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"]
Nov 02 01:33:37.130403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.128848    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-37b78d-openshift-default-dockercfg-mczbl\""
Nov 02 01:33:37.252333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.226041    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:37.252333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.242366    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7xk\" (UniqueName: \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-kube-api-access-7d7xk\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.252333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.242544    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-podinfo\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.252333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.242597    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-token\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.252333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.242619    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-data\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.252333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.242655    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-envoy\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.252333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.242794    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-credential-socket\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.252333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.242854    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-socket\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.252333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.242925    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-certs\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.252333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.242967    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istiod-ca-cert\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.389379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.376818    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:37.488959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.487086    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-podinfo\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.494464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.494416    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-token\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.494734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.494711    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-data\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.499324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.494916    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-envoy\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.525749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.525717    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-credential-socket\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.526062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.526038    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-socket\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.530648 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.530619    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-certs\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.530892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.530865    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istiod-ca-cert\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.557233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.526549    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-data\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.557233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.544709    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-certs\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.557233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.526054    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-credential-socket\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.557233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.526277    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-socket\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.557233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.545374    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istiod-ca-cert\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.557233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.555631    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-envoy\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.557967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.557940    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7xk\" (UniqueName: \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-kube-api-access-7d7xk\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.587550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.584348    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:37.588408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.588348    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-token\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.631139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.630635    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-podinfo\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.668913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.668884    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7xk\" (UniqueName: \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-kube-api-access-7d7xk\") pod \"e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") " pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.678955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.678927    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:37.931457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:37.931341    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:38.058415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:38.057879    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"]
Nov 02 01:33:38.064184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:38.064150    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86131d1-km25q"]
Nov 02 01:33:38.064401 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:33:38.064378    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602e7b84_294b_482d_afb8_fb2e5a9a37ce.slice/crio-4bc9b66cbcd3da0443c7422e0b80b6b0bc5108a1800e13e77d79c6d8e11695c5 WatchSource:0}: Error finding container 4bc9b66cbcd3da0443c7422e0b80b6b0bc5108a1800e13e77d79c6d8e11695c5: Status 404 returned error can't find the container with id 4bc9b66cbcd3da0443c7422e0b80b6b0bc5108a1800e13e77d79c6d8e11695c5
Nov 02 01:33:38.067386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:38.067352    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:38.067482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:38.067420    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:38.067482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:38.067448    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:38.071085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:38.070976    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"]
Nov 02 01:33:38.620909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:38.620876    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:38.725683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:38.725643    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" event={"ID":"602e7b84-294b-482d-afb8-fb2e5a9a37ce","Type":"ContainerStarted","Data":"29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9"}
Nov 02 01:33:38.725986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:38.725958    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" event={"ID":"602e7b84-294b-482d-afb8-fb2e5a9a37ce","Type":"ContainerStarted","Data":"4bc9b66cbcd3da0443c7422e0b80b6b0bc5108a1800e13e77d79c6d8e11695c5"}
Nov 02 01:33:38.849376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:38.849337    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b994aa4-ded5-4b3b-b825-dcb433421abe" path="/var/lib/kubelet/pods/3b994aa4-ded5-4b3b-b825-dcb433421abe/volumes"
Nov 02 01:33:39.338042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:39.337999    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:39.679635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:39.679606    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:39.685172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:39.685149    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:39.710191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:39.710117    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" podStartSLOduration=2.710103266 podStartE2EDuration="2.710103266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:38.776508992 +0000 UTC m=+1124.081441818" watchObservedRunningTime="2026-11-02 01:33:39.710103266 +0000 UTC m=+1125.015036091"
Nov 02 01:33:39.735868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:39.735841    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:39.736631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:39.736611    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:33:40.027526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:40.027449    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:40.835155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:40.835120    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:41.132569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:41.132476    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:41.266760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:41.266718    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:41.629906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:41.629871    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:42.332601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:42.332570    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:42.574460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:42.505732    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:43.260915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:43.260858    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:44.040511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:44.040479    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"]
Nov 02 01:33:44.047169 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:44.044944    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:44.072171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:44.072120    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1529a9a5-75mq5"]
Nov 02 01:33:44.739685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:44.739652    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:44.874041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:44.874005    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8352c9c4-9b10-4dd7-b632-970bc5e9c46f" path="/var/lib/kubelet/pods/8352c9c4-9b10-4dd7-b632-970bc5e9c46f/volumes"
Nov 02 01:33:45.503462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:45.503427    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:46.232251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:46.232207    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:46.645133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:46.645059    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:46.926490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:46.926460    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:47.656323 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:33:47.656270    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:33:47.705879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:47.705842    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:48.411412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:48.411381    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:49.100052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:49.100017    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:49.234316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:49.232956    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:50.199197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:50.199166    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:50.939179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:50.939144    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:51.720587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:51.720555    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:52.430902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:52.430863    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:53.108684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:53.108647    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:53.791081 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:53.791040    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:54.440929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:54.440880    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:55.234845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.234812    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:55.502714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.502629    4851 scope.go:122] "RemoveContainer" containerID="e76b5628c3c4126c7cd00d8220aa9fd806de6cc29087c3c27363b6b685aab198"
Nov 02 01:33:55.514085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.514052    4851 scope.go:122] "RemoveContainer" containerID="2abab2afc2a75b15953ffbc52c4d1966db9ec064b9c406575475aabb38180e55"
Nov 02 01:33:55.524908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.524879    4851 scope.go:122] "RemoveContainer" containerID="fc02b8ce3b478bab917ddd27179f6a56720bb6bdb29383fb2523eb5531bddecc"
Nov 02 01:33:55.535816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.535792    4851 scope.go:122] "RemoveContainer" containerID="e54503540621365c6ca6c3dd5d9dd81bf2090accc63ec03bf90194bcbe22b724"
Nov 02 01:33:55.546203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.546182    4851 scope.go:122] "RemoveContainer" containerID="9472bfb0e779f827120e3a8121c1009cf904e37d9420b8eedea2e81dfabed481"
Nov 02 01:33:55.559958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.559915    4851 scope.go:122] "RemoveContainer" containerID="44a720401aacaa71f069fc5364791eb4b36d203f1d4754bbb28b35c8551515de"
Nov 02 01:33:55.570841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.570821    4851 scope.go:122] "RemoveContainer" containerID="f72c452b60736f64524904c05720787a888699ee72adc77a70257b601574b715"
Nov 02 01:33:55.580847 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.580831    4851 scope.go:122] "RemoveContainer" containerID="3ad1e364da3a53413a6ce256f7b01e70d0e0dea39fda0c0b150924f41e7b9761"
Nov 02 01:33:55.590853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.590835    4851 scope.go:122] "RemoveContainer" containerID="455bea78cc04bfc96f915db9d3252f3c68377274213a2d50ff4a32400321edb0"
Nov 02 01:33:55.601177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.601156    4851 scope.go:122] "RemoveContainer" containerID="b714e9784bac376a2d975ac86f5898f87378effc2a6983259e67494be21dcd95"
Nov 02 01:33:55.611112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.611094    4851 scope.go:122] "RemoveContainer" containerID="4ed0e21e866e8dd1ed482ee26c55c33af3269feea86afb89e8d0aabaced167f7"
Nov 02 01:33:55.623179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.623149    4851 scope.go:122] "RemoveContainer" containerID="ac3cb2606798ebda5b09fa4af8629eb1694f1cb1c276d7f9aaeadcbc2d5cb109"
Nov 02 01:33:55.635591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.635572    4851 scope.go:122] "RemoveContainer" containerID="499c1f1a2cfdd76b96349214795f15b76a42d25e7bd22038ca937c5110d6ec53"
Nov 02 01:33:55.649217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.649176    4851 scope.go:122] "RemoveContainer" containerID="9ae04864daa16b05057acccf5673d4ee3eea21aa90e27640540009728ed03560"
Nov 02 01:33:55.660370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.660345    4851 scope.go:122] "RemoveContainer" containerID="af4cb02bad9a38ddce1ce1a4c966e07e4f674896c880e822fec51eacea6091b9"
Nov 02 01:33:55.671169 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.671147    4851 scope.go:122] "RemoveContainer" containerID="8fa5f3312e412de16a26dcd7dfd6080b8838c9a0476cae0dd036709e943c58aa"
Nov 02 01:33:55.681698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.681673    4851 scope.go:122] "RemoveContainer" containerID="e3879fe43d4aef00283c427ae26bac3a72c1b97d7f59b07d3d9c8926e4eac117"
Nov 02 01:33:55.692024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.692006    4851 scope.go:122] "RemoveContainer" containerID="a3e3cd5b42f2d3e11a15c1c6798a2d6224d1cd9504d88f434e630b2c8b0e5ccb"
Nov 02 01:33:55.702656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.702639    4851 scope.go:122] "RemoveContainer" containerID="309131c1b6aaec0a3dda09f7d27f35f764a761ef35369f44c86a191d6adc0b75"
Nov 02 01:33:55.716392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:55.716364    4851 scope.go:122] "RemoveContainer" containerID="ba21aa30fff6834276a5229759d96dcd59e71ff5e8a42f48768c4cff275fb2a5"
Nov 02 01:33:56.119223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:56.119133    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:56.771034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:56.771001    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:57.437120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:57.437087    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:58.245846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:58.245815    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:58.927980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:58.927942    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:33:59.062033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:59.061995    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"]
Nov 02 01:33:59.074640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:59.074589    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d3d1d0-c2rsh"]
Nov 02 01:33:59.841548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:33:59.841340    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:00.027344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:00.027271    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"]
Nov 02 01:34:00.032905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:00.032879    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cd804f-rf6jb"]
Nov 02 01:34:00.653808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:00.653775    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:00.848356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:00.848285    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d88eb42-dce9-4c57-adf8-a548eec1bedd" path="/var/lib/kubelet/pods/0d88eb42-dce9-4c57-adf8-a548eec1bedd/volumes"
Nov 02 01:34:00.848884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:00.848868    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9bb6b2-4311-4380-baaa-1f42d64a4aeb" path="/var/lib/kubelet/pods/bd9bb6b2-4311-4380-baaa-1f42d64a4aeb/volumes"
Nov 02 01:34:01.295312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:01.291340    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:01.424109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:01.424075    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:02.219074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:02.219043    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:02.919440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:02.919376    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:03.050037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:03.049973    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"]
Nov 02 01:34:03.063457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:03.060078    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9319d3bd-55nw4"]
Nov 02 01:34:03.681941 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:03.681903    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:04.543578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:04.543534    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:04.850579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:04.850505    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3119f15b-88ea-41ab-a4fa-c4a128051041" path="/var/lib/kubelet/pods/3119f15b-88ea-41ab-a4fa-c4a128051041/volumes"
Nov 02 01:34:05.032838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.032792    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"]
Nov 02 01:34:05.065320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.063041    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79741054-twmnq"]
Nov 02 01:34:05.389316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.371379    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"]
Nov 02 01:34:05.441404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.437525    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"]
Nov 02 01:34:05.441404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.437666    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:05.494687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.442390    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:34:05.494687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.442437    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:34:05.494687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.459823    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:34:05.494687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.459942    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:34:05.547857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.540162    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5k9\" (UniqueName: \"kubernetes.io/projected/8df77701-a09a-48e9-8a03-74237b3a79bb-kube-api-access-vr5k9\") pod \"maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:05.547857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.540257    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:05.547857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.540481    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:05.642609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.642490    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:05.643083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.643055    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5k9\" (UniqueName: \"kubernetes.io/projected/8df77701-a09a-48e9-8a03-74237b3a79bb-kube-api-access-vr5k9\") pod \"maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:05.643273 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.643255    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:05.645211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.645185    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:05.648530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.648507    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:05.659209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.659132    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5k9\" (UniqueName: \"kubernetes.io/projected/8df77701-a09a-48e9-8a03-74237b3a79bb-kube-api-access-vr5k9\") pod \"maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:05.804317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.803265    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:05.922735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:05.922706    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:06.776367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:06.758423    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:07.042916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:07.026369    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:07.575541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:07.574618    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:07.747969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:07.747932    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c35bed-145e-4b2c-88b2-0c5963d2df88" path="/var/lib/kubelet/pods/f6c35bed-145e-4b2c-88b2-0c5963d2df88/volumes"
Nov 02 01:34:07.813135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:07.813101    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"]
Nov 02 01:34:08.204317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:08.202784    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:08.332500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:08.332399    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:08.373498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:08.373452    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:08.520327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:08.513873    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:08.549333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:08.548455    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:08.606969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:08.605517    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh" event={"ID":"8df77701-a09a-48e9-8a03-74237b3a79bb","Type":"ContainerStarted","Data":"7c2f31474473b7638b63f9a19ef02f9426a4126dbee9540ddca5a44462bef57e"}
Nov 02 01:34:08.714607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:08.714466    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.034106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.033971    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:09.145337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.145187    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.146998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.145583    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:34:09.195316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.191371    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.377433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.377387    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.378593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.378569    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:34:09.409523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.404810    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:34:09.409523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.405262    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.410495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.410473    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:34:09.469752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.468406    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.469752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.468979    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.469752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.469482    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.518322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.517450    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:34:09.518322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.518104    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:34:09.582080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.582038    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.597327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.590949    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:34:09.776620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.764376    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.791681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.779827    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:09.791681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.781913    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:34:09.903111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:09.805894    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:10.158375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.155813    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:10.158572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.158501    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:10.290317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.277566    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:34:10.536767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.536733    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:34:10.545684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.414724    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:10.561096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.552322    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:34:10.580363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.416235    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:10.637317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.473463    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:10.637317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.473512    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:34:10.637317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.623851    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:34:10.637317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.623914    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:34:10.637317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.473538    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:34:10.637317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.299616    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:34:10.637317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.416266    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:10.682890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.596595    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:10.682890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.647777    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:34:10.682890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.648447    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:34:10.682890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.648496    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:34:10.682890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.659403    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:10.725646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.719853    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:10.781330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.780977    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:10.947059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:10.932439    4851 generic.go:356] "Generic (PLEG): container finished" podID="8df77701-a09a-48e9-8a03-74237b3a79bb" containerID="9dd27fe788844e2eabfc4e5673fefb2308156273f245539ec35c33adf6d2c88e" exitCode=0
Nov 02 01:34:11.160938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.156365    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:34:11.174361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.172832    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:11.339377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.331374    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh" event={"ID":"8df77701-a09a-48e9-8a03-74237b3a79bb","Type":"ContainerDied","Data":"9dd27fe788844e2eabfc4e5673fefb2308156273f245539ec35c33adf6d2c88e"}
Nov 02 01:34:11.390392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.390355    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:34:11.390697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.364165    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:11.555308 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.555242    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:11.575437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.575374    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:34:11.639435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.638745    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:11.641002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.640963    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:11.642511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.642483    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:11.690346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.654650    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:34:11.892379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.890840    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:34:11.892379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.663367    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:11.892379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.812008    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:11.892379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.663281    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:11.897713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:11.897686    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:34:12.027192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:12.025934    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:34:12.300567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:12.244279    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:34:12.300567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:12.244387    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:34:12.300567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:12.259020    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:34:12.393313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:12.374519    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:34:12.393313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:12.374630    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:34:12.393313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:12.374689    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:34:12.397053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:12.397019    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:12.458320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:12.430570    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:34:12.519458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:12.519429    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:34:13.018704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.014215    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:13.425410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.425364    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"]
Nov 02 01:34:13.493573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.493534    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930f803f-w5x55"]
Nov 02 01:34:13.511870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.511830    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh" event={"ID":"8df77701-a09a-48e9-8a03-74237b3a79bb","Type":"ContainerDied","Data":"7c2f31474473b7638b63f9a19ef02f9426a4126dbee9540ddca5a44462bef57e"}
Nov 02 01:34:13.512112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.512099    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c2f31474473b7638b63f9a19ef02f9426a4126dbee9540ddca5a44462bef57e"
Nov 02 01:34:13.528198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.528162    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:13.728620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.728527    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-xks-ca\") pod \"8df77701-a09a-48e9-8a03-74237b3a79bb\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") "
Nov 02 01:34:13.728797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.728771    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df77701-a09a-48e9-8a03-74237b3a79bb-kube-api-access-vr5k9\" (UniqueName: \"kubernetes.io/projected/8df77701-a09a-48e9-8a03-74237b3a79bb-kube-api-access-vr5k9\") pod \"8df77701-a09a-48e9-8a03-74237b3a79bb\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") "
Nov 02 01:34:13.729034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.729015    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-openshift-ca\") pod \"8df77701-a09a-48e9-8a03-74237b3a79bb\" (UID: \"8df77701-a09a-48e9-8a03-74237b3a79bb\") "
Nov 02 01:34:13.729553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.729524    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-openshift-ca" pod "8df77701-a09a-48e9-8a03-74237b3a79bb" (UID: "8df77701-a09a-48e9-8a03-74237b3a79bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:34:13.729792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.729771    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:13.731204 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.731180    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-xks-ca" pod "8df77701-a09a-48e9-8a03-74237b3a79bb" (UID: "8df77701-a09a-48e9-8a03-74237b3a79bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:34:13.756903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.756872    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df77701-a09a-48e9-8a03-74237b3a79bb-kube-api-access-vr5k9" pod "8df77701-a09a-48e9-8a03-74237b3a79bb" (UID: "8df77701-a09a-48e9-8a03-74237b3a79bb"). InnerVolumeSpecName "kube-api-access-vr5k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:34:13.808229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.808190    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:13.850333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.847580    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr5k9\" (UniqueName: \"kubernetes.io/projected/8df77701-a09a-48e9-8a03-74237b3a79bb-kube-api-access-vr5k9\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:13.850333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:13.847620    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df77701-a09a-48e9-8a03-74237b3a79bb-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:14.528276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:14.528238    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"
Nov 02 01:34:14.651313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:14.645188    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:14.853414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:14.853323    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80f66a8-8962-497c-ad8e-c6e8b4660713" path="/var/lib/kubelet/pods/b80f66a8-8962-497c-ad8e-c6e8b4660713/volumes"
Nov 02 01:34:15.034197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:15.034143    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"]
Nov 02 01:34:15.040941 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:15.040904    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"]
Nov 02 01:34:15.046808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:15.046778    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e82dc1-pq9zb"]
Nov 02 01:34:15.051906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:15.051880    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2721d457-vgtss"]
Nov 02 01:34:15.397183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:15.397091    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:15.749036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:15.748778    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"]
Nov 02 01:34:15.842476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:15.842391    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:15.968396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:15.968272    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:16.438515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:16.438432    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:16.887183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:16.882539    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c" path="/var/lib/kubelet/pods/028dec5f-8122-4d7e-8b8f-6cf9d04d6f4c/volumes"
Nov 02 01:34:16.887183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:16.883378    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f8a717-4e43-4d09-99a2-97d07bdad40d" path="/var/lib/kubelet/pods/52f8a717-4e43-4d09-99a2-97d07bdad40d/volumes"
Nov 02 01:34:17.045552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:17.045481    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"]
Nov 02 01:34:17.057067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:17.057024    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353d7753-8slxj"]
Nov 02 01:34:17.228029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:17.227974    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:17.879050 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:34:17.879010    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:34:17.946201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:17.946166    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:18.624161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:18.624122    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:18.850581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:18.850534    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d6724d-9aea-4f0e-96c8-24b977eff876" path="/var/lib/kubelet/pods/75d6724d-9aea-4f0e-96c8-24b977eff876/volumes"
Nov 02 01:34:19.339086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:19.339052    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:20.031068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:20.031016    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:20.712352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:20.712273    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:20.756729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:20.756682    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb" podUID="84e5798a-381e-4029-9b0a-364e77864ee5" containerName="payload-pre-processing" containerID="cri-o://6aaaac4332ebced9b215f5ca88b481e06d3a8258394c1225c9bcf758d01292e1" gracePeriod=25
Nov 02 01:34:21.404450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:21.404412    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:21.623614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:21.623576    4851 generic.go:356] "Generic (PLEG): container finished" podID="84e5798a-381e-4029-9b0a-364e77864ee5" containerID="6aaaac4332ebced9b215f5ca88b481e06d3a8258394c1225c9bcf758d01292e1" exitCode=0
Nov 02 01:34:21.623990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:21.623969    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb" event={"ID":"84e5798a-381e-4029-9b0a-364e77864ee5","Type":"ContainerDied","Data":"6aaaac4332ebced9b215f5ca88b481e06d3a8258394c1225c9bcf758d01292e1"}
Nov 02 01:34:21.839037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:21.839005    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:34:21.967080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:21.966998    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e5798a-381e-4029-9b0a-364e77864ee5-kube-api-access-82b2k\" (UniqueName: \"kubernetes.io/projected/84e5798a-381e-4029-9b0a-364e77864ee5-kube-api-access-82b2k\") pod \"84e5798a-381e-4029-9b0a-364e77864ee5\" (UID: \"84e5798a-381e-4029-9b0a-364e77864ee5\") "
Nov 02 01:34:21.967080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:21.967049    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e5798a-381e-4029-9b0a-364e77864ee5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e5798a-381e-4029-9b0a-364e77864ee5-plugins-config-volume\") pod \"84e5798a-381e-4029-9b0a-364e77864ee5\" (UID: \"84e5798a-381e-4029-9b0a-364e77864ee5\") "
Nov 02 01:34:21.967512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:21.967484    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e5798a-381e-4029-9b0a-364e77864ee5-plugins-config-volume" pod "84e5798a-381e-4029-9b0a-364e77864ee5" (UID: "84e5798a-381e-4029-9b0a-364e77864ee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:34:21.968153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:21.968129    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e5798a-381e-4029-9b0a-364e77864ee5-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:21.976424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:21.976394    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e5798a-381e-4029-9b0a-364e77864ee5-kube-api-access-82b2k" pod "84e5798a-381e-4029-9b0a-364e77864ee5" (UID: "84e5798a-381e-4029-9b0a-364e77864ee5"). InnerVolumeSpecName "kube-api-access-82b2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:34:22.069285 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:22.069252    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82b2k\" (UniqueName: \"kubernetes.io/projected/84e5798a-381e-4029-9b0a-364e77864ee5-kube-api-access-82b2k\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:22.121644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:22.121600    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:22.636096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:22.636068    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"
Nov 02 01:34:22.636459 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:22.636096    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb" event={"ID":"84e5798a-381e-4029-9b0a-364e77864ee5","Type":"ContainerDied","Data":"e6dec0fb4c644e8d3e7c69e040a2c7c249b6579f711e7df0b0334713f4276ed0"}
Nov 02 01:34:22.636459 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:22.636164    4851 scope.go:122] "RemoveContainer" containerID="6aaaac4332ebced9b215f5ca88b481e06d3a8258394c1225c9bcf758d01292e1"
Nov 02 01:34:22.664700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:22.664659    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"]
Nov 02 01:34:22.669843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:22.669810    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2ebe99-d795f9d95-54gwb"]
Nov 02 01:34:22.828973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:22.828927    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:22.924593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:22.922596    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e5798a-381e-4029-9b0a-364e77864ee5" path="/var/lib/kubelet/pods/84e5798a-381e-4029-9b0a-364e77864ee5/volumes"
Nov 02 01:34:23.529949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:23.529913    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:23.705514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:23.705407    4851 generic.go:356] "Generic (PLEG): container finished" podID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerID="e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029" exitCode=0
Nov 02 01:34:23.705514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:23.705482    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" event={"ID":"9b7f563e-5da1-4854-a8fa-e3931a462b3e","Type":"ContainerDied","Data":"e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029"}
Nov 02 01:34:23.706078 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:23.706057    4851 scope.go:122] "RemoveContainer" containerID="e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029"
Nov 02 01:34:23.707782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:23.707752    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:23.707869 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:23.707823    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:23.707869 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:23.707856    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:24.312340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:24.311989    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:24.353724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:24.353640    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:34:24.717976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:24.717940    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" event={"ID":"9b7f563e-5da1-4854-a8fa-e3931a462b3e","Type":"ContainerStarted","Data":"0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923"}
Nov 02 01:34:24.883996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:24.883944    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:25.538969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:25.537092    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:25.693899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:25.689825    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:34:25.697326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:25.695408    4851 patch_prober.go:29] interesting pod/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 01:34:25.697326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:25.695468    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 01:34:25.785468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:25.785432    4851 patch_prober.go:29] interesting pod/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:34:25.785684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:25.785491    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:34:26.352313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:26.344933    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:26.690641 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:26.690607    4851 patch_prober.go:29] interesting pod/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 01:34:26.691061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:26.690666    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 01:34:27.075373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.074743    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:27.603604 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.603563    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"]
Nov 02 01:34:27.604355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.604340    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8df77701-a09a-48e9-8a03-74237b3a79bb" containerName="revoke-keys"
Nov 02 01:34:27.604413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.604357    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df77701-a09a-48e9-8a03-74237b3a79bb" containerName="revoke-keys"
Nov 02 01:34:27.604413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.604379    4851 cpu_manager.go:414] "containerMap: removing container" podUID="84e5798a-381e-4029-9b0a-364e77864ee5" containerName="payload-pre-processing"
Nov 02 01:34:27.604413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.604385    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e5798a-381e-4029-9b0a-364e77864ee5" containerName="payload-pre-processing"
Nov 02 01:34:27.604517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.604485    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e5798a-381e-4029-9b0a-364e77864ee5" containerName="payload-pre-processing"
Nov 02 01:34:27.604517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.604505    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df77701-a09a-48e9-8a03-74237b3a79bb" containerName="revoke-keys"
Nov 02 01:34:27.607145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.607123    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:27.612251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.612214    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:34:27.612426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.612368    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:34:27.612493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.612430    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:34:27.615342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.612626    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:34:27.619669 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.619644    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"]
Nov 02 01:34:27.689643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.689602    4851 patch_prober.go:29] interesting pod/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 01:34:27.689838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.689667    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 01:34:27.731268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.731229    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzfsh\" (UniqueName: \"kubernetes.io/projected/aadab6a4-b16f-4ece-a644-480cfe0dd89b-kube-api-access-lzfsh\") pod \"maas-api-revoke-sub-da359c04-vhznb\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:27.731813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.731415    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da359c04-vhznb\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:27.731813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.731736    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da359c04-vhznb\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:27.811317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.805550    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:27.835672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.835635    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da359c04-vhznb\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:27.865358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.865248    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzfsh\" (UniqueName: \"kubernetes.io/projected/aadab6a4-b16f-4ece-a644-480cfe0dd89b-kube-api-access-lzfsh\") pod \"maas-api-revoke-sub-da359c04-vhznb\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:27.875311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.865686    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da359c04-vhznb\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:27.875986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.853184    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da359c04-vhznb\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:27.876173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.867655    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da359c04-vhznb\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:27.911708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.901462    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzfsh\" (UniqueName: \"kubernetes.io/projected/aadab6a4-b16f-4ece-a644-480cfe0dd89b-kube-api-access-lzfsh\") pod \"maas-api-revoke-sub-da359c04-vhznb\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:27.920974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.920943    4851 generic.go:356] "Generic (PLEG): container finished" podID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerID="29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9" exitCode=0
Nov 02 01:34:27.921186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.921172    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" event={"ID":"602e7b84-294b-482d-afb8-fb2e5a9a37ce","Type":"ContainerDied","Data":"29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9"}
Nov 02 01:34:27.921736 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.921720    4851 scope.go:122] "RemoveContainer" containerID="29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9"
Nov 02 01:34:27.923544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.923511    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:27.923658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.923585    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:27.923658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.923618    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:27.924483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:27.924464    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:28.323490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:28.323457    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"]
Nov 02 01:34:28.325438 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:34:28.325383    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadab6a4_b16f_4ece_a644_480cfe0dd89b.slice/crio-7cf683b2172e31d7d9a95682c18a5e223020c5a425a6afadd32fec516aa490af WatchSource:0}: Error finding container 7cf683b2172e31d7d9a95682c18a5e223020c5a425a6afadd32fec516aa490af: Status 404 returned error can't find the container with id 7cf683b2172e31d7d9a95682c18a5e223020c5a425a6afadd32fec516aa490af
Nov 02 01:34:28.412327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:28.412280    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:28.690396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:28.690358    4851 patch_prober.go:29] interesting pod/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 01:34:28.690562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:28.690405    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 01:34:28.931918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:28.931880    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" event={"ID":"602e7b84-294b-482d-afb8-fb2e5a9a37ce","Type":"ContainerStarted","Data":"86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3"}
Nov 02 01:34:28.962318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:28.957938    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb" event={"ID":"aadab6a4-b16f-4ece-a644-480cfe0dd89b","Type":"ContainerStarted","Data":"2d82e41dcd9e8a80458b853cda2fbcafda772d84bfae81a1410f36da7ea56ad6"}
Nov 02 01:34:28.962318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:28.958002    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb" event={"ID":"aadab6a4-b16f-4ece-a644-480cfe0dd89b","Type":"ContainerStarted","Data":"7cf683b2172e31d7d9a95682c18a5e223020c5a425a6afadd32fec516aa490af"}
Nov 02 01:34:29.007596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:29.007452    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb" podStartSLOduration=2.007437032 podStartE2EDuration="2.007437032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:34:29.001539417 +0000 UTC m=+1174.306472242" watchObservedRunningTime="2026-11-02 01:34:29.007437032 +0000 UTC m=+1174.312369879"
Nov 02 01:34:29.140315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:29.137821    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:29.685351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:29.684700    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:34:29.685351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:29.685033    4851 patch_prober.go:29] interesting pod/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 01:34:29.686598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:29.686452    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 01:34:29.690473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:29.689654    4851 patch_prober.go:29] interesting pod/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 01:34:29.690473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:29.689706    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 01:34:29.822346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:29.822272    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:30.045314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.044641    4851 generic.go:356] "Generic (PLEG): container finished" podID="aadab6a4-b16f-4ece-a644-480cfe0dd89b" containerID="2d82e41dcd9e8a80458b853cda2fbcafda772d84bfae81a1410f36da7ea56ad6" exitCode=0
Nov 02 01:34:30.058315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.046190    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb" event={"ID":"aadab6a4-b16f-4ece-a644-480cfe0dd89b","Type":"ContainerDied","Data":"2d82e41dcd9e8a80458b853cda2fbcafda772d84bfae81a1410f36da7ea56ad6"}
Nov 02 01:34:30.077411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.077380    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"]
Nov 02 01:34:30.096338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.095343    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e13dc1b-zxzkz"]
Nov 02 01:34:30.519044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.519010    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:30.682000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.681964    4851 patch_prober.go:29] interesting pod/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 01:34:30.682199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.682021    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 01:34:30.718345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.717856    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:34:30.718980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.718961    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:34:30.720013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.719989    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:34:30.872713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:30.872634    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ae183c-f86b-4560-adca-9430c3f39eeb" path="/var/lib/kubelet/pods/b0ae183c-f86b-4560-adca-9430c3f39eeb/volumes"
Nov 02 01:34:31.207619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.207581    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:31.450316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.445430    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:31.657377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.656943    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadab6a4-b16f-4ece-a644-480cfe0dd89b-kube-api-access-lzfsh\" (UniqueName: \"kubernetes.io/projected/aadab6a4-b16f-4ece-a644-480cfe0dd89b-kube-api-access-lzfsh\") pod \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") "
Nov 02 01:34:31.657377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.657053    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-xks-ca\") pod \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") "
Nov 02 01:34:31.657377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.657130    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-openshift-ca\") pod \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\" (UID: \"aadab6a4-b16f-4ece-a644-480cfe0dd89b\") "
Nov 02 01:34:31.658454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.658430    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-openshift-ca" pod "aadab6a4-b16f-4ece-a644-480cfe0dd89b" (UID: "aadab6a4-b16f-4ece-a644-480cfe0dd89b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:34:31.660832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.660801    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-xks-ca" pod "aadab6a4-b16f-4ece-a644-480cfe0dd89b" (UID: "aadab6a4-b16f-4ece-a644-480cfe0dd89b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:34:31.660832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.660826    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadab6a4-b16f-4ece-a644-480cfe0dd89b-kube-api-access-lzfsh" pod "aadab6a4-b16f-4ece-a644-480cfe0dd89b" (UID: "aadab6a4-b16f-4ece-a644-480cfe0dd89b"). InnerVolumeSpecName "kube-api-access-lzfsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:34:31.680759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.680728    4851 patch_prober.go:29] interesting pod/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 01:34:31.680861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.680796    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 01:34:31.759395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.759365    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:31.759395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.759391    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzfsh\" (UniqueName: \"kubernetes.io/projected/aadab6a4-b16f-4ece-a644-480cfe0dd89b-kube-api-access-lzfsh\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:31.759654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.759407    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aadab6a4-b16f-4ece-a644-480cfe0dd89b-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:31.960895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:31.960860    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:32.118394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.117156    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"
Nov 02 01:34:32.118394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.117342    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb" event={"ID":"aadab6a4-b16f-4ece-a644-480cfe0dd89b","Type":"ContainerDied","Data":"7cf683b2172e31d7d9a95682c18a5e223020c5a425a6afadd32fec516aa490af"}
Nov 02 01:34:32.118394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.117402    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cf683b2172e31d7d9a95682c18a5e223020c5a425a6afadd32fec516aa490af"
Nov 02 01:34:32.122987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.122853    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"]
Nov 02 01:34:32.127883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.127860    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"]
Nov 02 01:34:32.138318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.132925    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adc04e2-bg9ww"]
Nov 02 01:34:32.149496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.146491    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6407b46-cmsk7"]
Nov 02 01:34:32.623606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.623565    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:32.685315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.684697    4851 patch_prober.go:29] interesting pod/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 01:34:32.685315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.684772    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 01:34:32.866378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.866346    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362c6a8e-907e-4aae-b272-412bbe0c848b" path="/var/lib/kubelet/pods/362c6a8e-907e-4aae-b272-412bbe0c848b/volumes"
Nov 02 01:34:32.866878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:32.866862    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a39c85-e039-4c53-896d-f81e4754917c" path="/var/lib/kubelet/pods/c5a39c85-e039-4c53-896d-f81e4754917c/volumes"
Nov 02 01:34:33.042471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:33.042412    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"]
Nov 02 01:34:33.068994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:33.068955    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6810f982-sx8vj"]
Nov 02 01:34:33.449448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:33.447813    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:33.680675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:33.680644    4851 patch_prober.go:29] interesting pod/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 01:34:33.681063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:33.680701    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 01:34:34.025374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:34.025343    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"]
Nov 02 01:34:34.030220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:34.030194    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e128ad-twpqk"]
Nov 02 01:34:34.210437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:34.210396    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:34.713664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:34.713624    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:34:34.714335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:34.714202    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:34:34.714995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:34.714968    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:34:34.848741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:34.848709    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29356a60-2eeb-438b-a57b-a589b3faf0f8" path="/var/lib/kubelet/pods/29356a60-2eeb-438b-a57b-a589b3faf0f8/volumes"
Nov 02 01:34:34.849341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:34.849318    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a2d739-07d2-437e-bd57-806e0525ad24" path="/var/lib/kubelet/pods/c7a2d739-07d2-437e-bd57-806e0525ad24/volumes"
Nov 02 01:34:35.011500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:35.011382    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:35.761729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:35.761696    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:36.436970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:36.436939    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:37.124965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:37.124931    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:37.671483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:37.671445    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:37.850854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:37.850815    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:38.002363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.000851    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:38.057315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.055671    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"]
Nov 02 01:34:38.128317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.124611    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"]
Nov 02 01:34:38.145124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.145081    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0e38cb-zks4t"]
Nov 02 01:34:38.183375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.183341    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d4d948-qzqt8"]
Nov 02 01:34:38.188757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.188716    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"]
Nov 02 01:34:38.279082 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.278883    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:38.289319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.284788    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ebe99-wzzsh"]
Nov 02 01:34:38.406417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.406379    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:38.692046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.692016    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:38.849881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.849840    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df77701-a09a-48e9-8a03-74237b3a79bb" path="/var/lib/kubelet/pods/8df77701-a09a-48e9-8a03-74237b3a79bb/volumes"
Nov 02 01:34:38.850437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.850421    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d77d12-7108-4358-91f7-08057e46435c" path="/var/lib/kubelet/pods/a8d77d12-7108-4358-91f7-08057e46435c/volumes"
Nov 02 01:34:38.850894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:38.850880    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eacc2ba5-4848-44bb-8253-e4e0ae9c1300" path="/var/lib/kubelet/pods/eacc2ba5-4848-44bb-8253-e4e0ae9c1300/volumes"
Nov 02 01:34:39.081089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:39.080985    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:39.735099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:39.735049    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:40.026449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:40.026361    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"]
Nov 02 01:34:40.030966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:40.030940    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890c7d8b-x2s2k"]
Nov 02 01:34:40.452135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:40.452103    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:40.850650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:40.850546    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb3ed0ec-2347-4765-982b-28e52a7d1fd1" path="/var/lib/kubelet/pods/eb3ed0ec-2347-4765-982b-28e52a7d1fd1/volumes"
Nov 02 01:34:41.413318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:41.403821    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:41.877616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:41.877527    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:42.251202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.251169    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"]
Nov 02 01:34:42.251948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.251919    4851 cpu_manager.go:414] "containerMap: removing container" podUID="aadab6a4-b16f-4ece-a644-480cfe0dd89b" containerName="revoke-keys"
Nov 02 01:34:42.251948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.251940    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aadab6a4-b16f-4ece-a644-480cfe0dd89b" containerName="revoke-keys"
Nov 02 01:34:42.252155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.252095    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="aadab6a4-b16f-4ece-a644-480cfe0dd89b" containerName="revoke-keys"
Nov 02 01:34:42.256176 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.256150    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.273183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.273154    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-677312-openshift-default-dockercfg-q9hkc\""
Nov 02 01:34:42.274429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.274403    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"]
Nov 02 01:34:42.318229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.318191    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:42.382621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.382589    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e310f6b-1d42-4471-a2a2-3562f8199803-istiod-ca-cert\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.382621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.382631    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rhf\" (UniqueName: \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-kube-api-access-r7rhf\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.382924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.382723    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-credential-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.382924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.382853    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-envoy\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.383009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.382981    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-data\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.383065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.383051    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-certs\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.383116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.383081    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.383153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.383135    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-token\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.383216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.383205    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-podinfo\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.472322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.468383    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:42.484976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.484929    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-data\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.495317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.485828    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-certs\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.495737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.495710    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.495940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.495925    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-token\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.496121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.496104    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-podinfo\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.496427 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.496409    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e310f6b-1d42-4471-a2a2-3562f8199803-istiod-ca-cert\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.496573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.496557    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rhf\" (UniqueName: \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-kube-api-access-r7rhf\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.496668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.496641    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.496730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.485700    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-data\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.496876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.496859    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-credential-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.497020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.494931    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-certs\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.497088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.497000    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-envoy\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.506402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.497913    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e310f6b-1d42-4471-a2a2-3562f8199803-istiod-ca-cert\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.506402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.498939    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-credential-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.565449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.565419    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-podinfo\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.565965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.565947    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-envoy\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.609315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.609265    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rhf\" (UniqueName: \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-kube-api-access-r7rhf\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.610612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.610584    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-token\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:42.876156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:42.876055    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:43.184511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:43.184484    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"]
Nov 02 01:34:43.185053 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:34:43.185026    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e310f6b_1d42_4471_a2a2_3562f8199803.slice/crio-7356af1585d8c69397c0423c8bc775a3f1e1d9974c89c88d56524525ed9ebcd2 WatchSource:0}: Error finding container 7356af1585d8c69397c0423c8bc775a3f1e1d9974c89c88d56524525ed9ebcd2: Status 404 returned error can't find the container with id 7356af1585d8c69397c0423c8bc775a3f1e1d9974c89c88d56524525ed9ebcd2
Nov 02 01:34:43.188491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:43.188449    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:43.188599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:43.188544    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:43.188599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:43.188588    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:43.240678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:43.240615    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:43.384102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:43.383600    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" event={"ID":"5e310f6b-1d42-4471-a2a2-3562f8199803","Type":"ContainerStarted","Data":"7356af1585d8c69397c0423c8bc775a3f1e1d9974c89c88d56524525ed9ebcd2"}
Nov 02 01:34:43.922634 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:43.922587    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:44.054469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:44.054418    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"]
Nov 02 01:34:44.057116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:44.057083    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07ff9fb7-4r9rn"]
Nov 02 01:34:44.395709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:44.395602    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" event={"ID":"5e310f6b-1d42-4471-a2a2-3562f8199803","Type":"ContainerStarted","Data":"cc3967322c448e22c1ec6f9ce1284315850dfe653741ac4d0aa355da5cbc713b"}
Nov 02 01:34:44.421791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:44.421740    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" podStartSLOduration=2.421726471 podStartE2EDuration="2.421726471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:34:44.419072807 +0000 UTC m=+1189.724005634" watchObservedRunningTime="2026-11-02 01:34:44.421726471 +0000 UTC m=+1189.726659373"
Nov 02 01:34:44.730214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:44.730149    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:44.848548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:44.848512    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763c53db-cc32-40fe-a758-67b9d7ccbef9" path="/var/lib/kubelet/pods/763c53db-cc32-40fe-a758-67b9d7ccbef9/volumes"
Nov 02 01:34:44.868953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:44.868918    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:44.876978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:44.876951    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:44.883431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:44.883407    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:45.406139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:45.406105    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:45.407429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:45.407409    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:34:45.544355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:45.544276    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:45.941001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:45.940949    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:46.078563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.078532    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:46.264407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.263943    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"]
Nov 02 01:34:46.264407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.264212    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy" containerID="cri-o://0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923" gracePeriod=30
Nov 02 01:34:46.316725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.316695    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:46.463318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.459208    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:46.818888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.818848    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"]
Nov 02 01:34:46.825080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.825057    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:46.830181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.830155    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:34:46.830347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.830244    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:34:46.835232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.835209    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"]
Nov 02 01:34:46.880906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.880878    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:46.894953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.894927    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:34:46.895261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.895241    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:34:46.983843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.983810    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b78d-58pmr\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:46.984201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.984178    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz74v\" (UniqueName: \"kubernetes.io/projected/5eeabe40-8874-476e-ad67-f7e0d8f78109-kube-api-access-zz74v\") pod \"maas-api-revoke-keys-e2e-api-b-37b78d-58pmr\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:46.984399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:46.984382    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b78d-58pmr\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:47.085867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.085772    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz74v\" (UniqueName: \"kubernetes.io/projected/5eeabe40-8874-476e-ad67-f7e0d8f78109-kube-api-access-zz74v\") pod \"maas-api-revoke-keys-e2e-api-b-37b78d-58pmr\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:47.086043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.085947    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b78d-58pmr\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:47.086085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.086039    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b78d-58pmr\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:47.086728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.086701    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b78d-58pmr\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:47.088583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.088565    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-37b78d-58pmr\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:47.098244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.098221    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz74v\" (UniqueName: \"kubernetes.io/projected/5eeabe40-8874-476e-ad67-f7e0d8f78109-kube-api-access-zz74v\") pod \"maas-api-revoke-keys-e2e-api-b-37b78d-58pmr\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:47.144217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.144180    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:47.320898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.318756    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:47.501890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.500426    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"]
Nov 02 01:34:47.503075 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:34:47.503044    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eeabe40_8874_476e_ad67_f7e0d8f78109.slice/crio-c5547d0a34e09f1550453d174e1937274ac356c6dc4318356f7254ba554cd028 WatchSource:0}: Error finding container c5547d0a34e09f1550453d174e1937274ac356c6dc4318356f7254ba554cd028: Status 404 returned error can't find the container with id c5547d0a34e09f1550453d174e1937274ac356c6dc4318356f7254ba554cd028
Nov 02 01:34:47.544347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.544317    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr" event={"ID":"5eeabe40-8874-476e-ad67-f7e0d8f78109","Type":"ContainerStarted","Data":"c5547d0a34e09f1550453d174e1937274ac356c6dc4318356f7254ba554cd028"}
Nov 02 01:34:47.660403 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:34:47.660370    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:34:47.817555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.817468    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:47.985952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:47.985918    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:48.336465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:48.336430    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:48.556000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:48.555968    4851 generic.go:356] "Generic (PLEG): container finished" podID="5eeabe40-8874-476e-ad67-f7e0d8f78109" containerID="10f742f0710f4b23c9c9b79b8b38ebdbbaad1c0913a095e422bb55578a70cedb" exitCode=0
Nov 02 01:34:48.556444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:48.556062    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr" event={"ID":"5eeabe40-8874-476e-ad67-f7e0d8f78109","Type":"ContainerDied","Data":"10f742f0710f4b23c9c9b79b8b38ebdbbaad1c0913a095e422bb55578a70cedb"}
Nov 02 01:34:49.031978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.031948    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:49.214381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.199189    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:49.789467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.789440    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:49.805803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.805775    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eeabe40-8874-476e-ad67-f7e0d8f78109-kube-api-access-zz74v\" (UniqueName: \"kubernetes.io/projected/5eeabe40-8874-476e-ad67-f7e0d8f78109-kube-api-access-zz74v\") pod \"5eeabe40-8874-476e-ad67-f7e0d8f78109\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") "
Nov 02 01:34:49.805929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.805828    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-xks-ca\") pod \"5eeabe40-8874-476e-ad67-f7e0d8f78109\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") "
Nov 02 01:34:49.805929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.805915    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-openshift-ca\") pod \"5eeabe40-8874-476e-ad67-f7e0d8f78109\" (UID: \"5eeabe40-8874-476e-ad67-f7e0d8f78109\") "
Nov 02 01:34:49.806237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.806215    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-openshift-ca" pod "5eeabe40-8874-476e-ad67-f7e0d8f78109" (UID: "5eeabe40-8874-476e-ad67-f7e0d8f78109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:34:49.806637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.806610    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:49.808580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.808558    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-xks-ca" pod "5eeabe40-8874-476e-ad67-f7e0d8f78109" (UID: "5eeabe40-8874-476e-ad67-f7e0d8f78109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:34:49.809339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.809266    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eeabe40-8874-476e-ad67-f7e0d8f78109-kube-api-access-zz74v" pod "5eeabe40-8874-476e-ad67-f7e0d8f78109" (UID: "5eeabe40-8874-476e-ad67-f7e0d8f78109"). InnerVolumeSpecName "kube-api-access-zz74v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:34:49.908462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.908383    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz74v\" (UniqueName: \"kubernetes.io/projected/5eeabe40-8874-476e-ad67-f7e0d8f78109-kube-api-access-zz74v\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:49.908462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.908408    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeabe40-8874-476e-ad67-f7e0d8f78109-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:49.975389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:49.975360    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:50.209170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:50.209135    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:50.452394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:50.452357    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:50.624360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:50.624243    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"
Nov 02 01:34:50.624360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:50.624264    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr" event={"ID":"5eeabe40-8874-476e-ad67-f7e0d8f78109","Type":"ContainerDied","Data":"c5547d0a34e09f1550453d174e1937274ac356c6dc4318356f7254ba554cd028"}
Nov 02 01:34:50.624565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:50.624367    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5547d0a34e09f1550453d174e1937274ac356c6dc4318356f7254ba554cd028"
Nov 02 01:34:50.721318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:50.714075    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:51.004919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.004854    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:51.231833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.231800    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:51.435242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.435211    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:51.671165 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.671101    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:51.687135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.687099    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:34:51.692145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.692118    4851 generic.go:356] "Generic (PLEG): container finished" podID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerID="0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923" exitCode=0
Nov 02 01:34:51.692350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.692169    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"
Nov 02 01:34:51.692350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.692194    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" event={"ID":"9b7f563e-5da1-4854-a8fa-e3931a462b3e","Type":"ContainerDied","Data":"0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923"}
Nov 02 01:34:51.692350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.692264    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln" event={"ID":"9b7f563e-5da1-4854-a8fa-e3931a462b3e","Type":"ContainerDied","Data":"17e654075638d73bc06681928d54d54557b8814b41abb47d4ae72b3d89770abf"}
Nov 02 01:34:51.692350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.692302    4851 scope.go:122] "RemoveContainer" containerID="0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923"
Nov 02 01:34:51.704542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.704522    4851 scope.go:122] "RemoveContainer" containerID="e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029"
Nov 02 01:34:51.716013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.715979    4851 scope.go:122] "RemoveContainer" containerID="0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923"
Nov 02 01:34:51.716372 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:34:51.716314    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923\": container with ID starting with 0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923 not found: ID does not exist" containerID="0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923"
Nov 02 01:34:51.716372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.716346    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923"} err="failed to get container status \"0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923\": rpc error: code = NotFound desc = could not find container \"0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923\": container with ID starting with 0778eddd47362ef050cca40e058da6d72903fe026130289bed2ed242cb308923 not found: ID does not exist"
Nov 02 01:34:51.716372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.716370    4851 scope.go:122] "RemoveContainer" containerID="e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029"
Nov 02 01:34:51.716680 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:34:51.716661    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029\": container with ID starting with e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029 not found: ID does not exist" containerID="e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029"
Nov 02 01:34:51.716722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.716699    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029"} err="failed to get container status \"e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029\": rpc error: code = NotFound desc = could not find container \"e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029\": container with ID starting with e2e3c114b5a26894b6ff2db1fad8323b0bfcae949fae35a2820a2d23d5e74029 not found: ID does not exist"
Nov 02 01:34:51.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818137    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-podinfo\") pod \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") "
Nov 02 01:34:51.818228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818202    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-token\" (UniqueName: \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-token\") pod \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") "
Nov 02 01:34:51.818513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818329    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-data\") pod \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") "
Nov 02 01:34:51.818513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818430    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istiod-ca-cert\") pod \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") "
Nov 02 01:34:51.818612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818573    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-data" pod "9b7f563e-5da1-4854-a8fa-e3931a462b3e" (UID: "9b7f563e-5da1-4854-a8fa-e3931a462b3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:34:51.818612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818582    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-kube-api-access-spnb4\" (UniqueName: \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-kube-api-access-spnb4\") pod \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") "
Nov 02 01:34:51.818722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818704    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-envoy\") pod \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") "
Nov 02 01:34:51.818879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818809    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-socket\") pod \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") "
Nov 02 01:34:51.818879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818817    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istiod-ca-cert" pod "9b7f563e-5da1-4854-a8fa-e3931a462b3e" (UID: "9b7f563e-5da1-4854-a8fa-e3931a462b3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:34:51.819337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818920    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-credential-socket\") pod \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") "
Nov 02 01:34:51.819337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.818958    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-certs\") pod \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\" (UID: \"9b7f563e-5da1-4854-a8fa-e3931a462b3e\") "
Nov 02 01:34:51.819337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.819168    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-socket" pod "9b7f563e-5da1-4854-a8fa-e3931a462b3e" (UID: "9b7f563e-5da1-4854-a8fa-e3931a462b3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:34:51.819337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.819329    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-certs" pod "9b7f563e-5da1-4854-a8fa-e3931a462b3e" (UID: "9b7f563e-5da1-4854-a8fa-e3931a462b3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:34:51.819583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.819380    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-credential-socket" pod "9b7f563e-5da1-4854-a8fa-e3931a462b3e" (UID: "9b7f563e-5da1-4854-a8fa-e3931a462b3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:34:51.819896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.819871    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:51.819978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.819902    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:51.819978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.819917    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:51.819978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.819932    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:51.819978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.819945    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:51.821017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.820992    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-token" pod "9b7f563e-5da1-4854-a8fa-e3931a462b3e" (UID: "9b7f563e-5da1-4854-a8fa-e3931a462b3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:34:51.821458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.821274    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-kube-api-access-spnb4" pod "9b7f563e-5da1-4854-a8fa-e3931a462b3e" (UID: "9b7f563e-5da1-4854-a8fa-e3931a462b3e"). InnerVolumeSpecName "kube-api-access-spnb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:34:51.821458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.821389    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-envoy" pod "9b7f563e-5da1-4854-a8fa-e3931a462b3e" (UID: "9b7f563e-5da1-4854-a8fa-e3931a462b3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:34:51.821884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.821866    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-podinfo" pod "9b7f563e-5da1-4854-a8fa-e3931a462b3e" (UID: "9b7f563e-5da1-4854-a8fa-e3931a462b3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:34:51.921248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.921220    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:51.921248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.921248    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:51.921466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.921257    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spnb4\" (UniqueName: \"kubernetes.io/projected/9b7f563e-5da1-4854-a8fa-e3931a462b3e-kube-api-access-spnb4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:51.921466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:51.921267    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b7f563e-5da1-4854-a8fa-e3931a462b3e-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:34:52.008303 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.008268    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:52.021068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.021038    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"]
Nov 02 01:34:52.024824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.024801    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2ebe99-openshift-default-596fd44597-wgzln"]
Nov 02 01:34:52.235892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.235861    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"]
Nov 02 01:34:52.236629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.236602    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy"
Nov 02 01:34:52.236629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.236623    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy"
Nov 02 01:34:52.236842 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.236662    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5eeabe40-8874-476e-ad67-f7e0d8f78109" containerName="revoke-keys"
Nov 02 01:34:52.236842 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.236670    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eeabe40-8874-476e-ad67-f7e0d8f78109" containerName="revoke-keys"
Nov 02 01:34:52.236950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.236894    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eeabe40-8874-476e-ad67-f7e0d8f78109" containerName="revoke-keys"
Nov 02 01:34:52.236950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.236915    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy"
Nov 02 01:34:52.236950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.236935    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy"
Nov 02 01:34:52.247512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.247467    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.253913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.253874    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-bbb385-openshift-default-dockercfg-zgx6l\""
Nov 02 01:34:52.255106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.255083    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"]
Nov 02 01:34:52.284765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.284736    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:52.435678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.435634    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-certs\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.457993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.457961    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-credential-socket\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.458332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.458307    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-socket\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.458770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.458736    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-istio-token\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.459052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.459029    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsr4v\" (UniqueName: \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-kube-api-access-qsr4v\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.459223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.459208    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-envoy\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.459393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.459378    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70565ca1-c120-4b77-b307-a16604644193-istio-podinfo\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.459580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.459564    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70565ca1-c120-4b77-b307-a16604644193-istiod-ca-cert\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.459828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.459814    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-data\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.525371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.524322    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:52.563315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.562509    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-data\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.563315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.562662    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-certs\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.563315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.562755    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-credential-socket\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.563315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.562785    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-socket\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.563315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.562943    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-istio-token\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.563315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.563001    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsr4v\" (UniqueName: \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-kube-api-access-qsr4v\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.563315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.563031    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-envoy\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.563315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.563058    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70565ca1-c120-4b77-b307-a16604644193-istio-podinfo\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.563315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.563107    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70565ca1-c120-4b77-b307-a16604644193-istiod-ca-cert\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.564616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.563919    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70565ca1-c120-4b77-b307-a16604644193-istiod-ca-cert\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.564616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.564248    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-data\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.564616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.564543    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-certs\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.601896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.564769    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-credential-socket\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.601896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.564991    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-socket\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.601896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.596196    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70565ca1-c120-4b77-b307-a16604644193-istio-podinfo\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.614070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.613951    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsr4v\" (UniqueName: \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-kube-api-access-qsr4v\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.633366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.618723    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-envoy\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.645372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.638814    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-istio-token\") pod \"e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") " pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:52.766492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.766305    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:52.850858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.850778    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" path="/var/lib/kubelet/pods/9b7f563e-5da1-4854-a8fa-e3931a462b3e/volumes"
Nov 02 01:34:52.870379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:52.870354    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:53.092913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:53.092874    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:53.218316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:53.218166    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"]
Nov 02 01:34:53.240483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:53.224879    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:53.240483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:53.231594    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:53.240483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:53.231697    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:34:53.753063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:53.753025    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9" event={"ID":"70565ca1-c120-4b77-b307-a16604644193","Type":"ContainerStarted","Data":"94fc71b5e91b0a2ba607880b88ac0425b6eb41e9276a83763849a931901ce78f"}
Nov 02 01:34:53.753243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:53.753086    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9" event={"ID":"70565ca1-c120-4b77-b307-a16604644193","Type":"ContainerStarted","Data":"ef27e4f84a08eaaa81e1b5015e6ac7a095a95c5e9a221420342cf84c09a460c7"}
Nov 02 01:34:53.784073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:53.784023    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9" podStartSLOduration=1.7840107459999999 podStartE2EDuration="1.784010746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:34:53.780552566 +0000 UTC m=+1199.085485394" watchObservedRunningTime="2026-11-02 01:34:53.784010746 +0000 UTC m=+1199.088943642"
Nov 02 01:34:53.807020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:53.806991    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:53.871058 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:53.871028    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:54.081367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:54.078183    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:54.163317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:54.151153    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"]
Nov 02 01:34:54.218386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:54.218342    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"]
Nov 02 01:34:54.274573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:54.260175    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3978ac42-qg62n"]
Nov 02 01:34:54.296315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:54.284409    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78b341c-7p7b8"]
Nov 02 01:34:54.303319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:54.299771    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:54.705928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:54.705898    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:55.004969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.004835    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f074b2-0dca-42a3-9eba-2b1480122136" path="/var/lib/kubelet/pods/85f074b2-0dca-42a3-9eba-2b1480122136/volumes"
Nov 02 01:34:55.006194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.006153    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d280a75e-6351-4e05-b522-cbb429cffa85" path="/var/lib/kubelet/pods/d280a75e-6351-4e05-b522-cbb429cffa85/volumes"
Nov 02 01:34:55.008935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.008915    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:55.009914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.009880    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:55.162353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.162325    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:34:55.163135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.163118    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:34:55.770908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.770880    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:55.771968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.771951    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:34:55.776645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.776615    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:55.871933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.871907    4851 scope.go:122] "RemoveContainer" containerID="a8b4033335cd2b045df811d69b41657d2a34563a9d39c47626cf67351ec09e76"
Nov 02 01:34:55.883717 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.883691    4851 scope.go:122] "RemoveContainer" containerID="bd5d4a9471dc78f9dfac5e7c6816db0451bcb45d76d272762273c306d48b4ba1"
Nov 02 01:34:55.895768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.895746    4851 scope.go:122] "RemoveContainer" containerID="1affcd6ba679a12640de37d645b7702b13f0884aa4b5431ddb9bd9c9703470c2"
Nov 02 01:34:55.965213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:55.963357    4851 scope.go:122] "RemoveContainer" containerID="51037fc6f21730927d0a292d964120352aa2a39ebbe1bd1bdc884cceef7475e7"
Nov 02 01:34:56.098055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:56.098029    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:56.437696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:56.437662    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:56.795218 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:56.795129    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:57.207422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:57.207386    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:57.426312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:57.419938    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:57.890857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:57.890820    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:57.970478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:57.970443    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"]
Nov 02 01:34:57.971201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:57.971178    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy"
Nov 02 01:34:57.971201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:57.971200    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7f563e-5da1-4854-a8fa-e3931a462b3e" containerName="istio-proxy"
Nov 02 01:34:57.977929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:57.977900    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:34:57.988305 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:57.988258    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-bbb385\""
Nov 02 01:34:57.989766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:57.989737    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"]
Nov 02 01:34:58.048100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.048062    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"]
Nov 02 01:34:58.053031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.053004    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:34:58.058154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.058116    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-bbb385-dockercfg-m8qcc\""
Nov 02 01:34:58.065873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.065838    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"]
Nov 02 01:34:58.148461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.137527    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:58.198449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.195962    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtgb\" (UniqueName: \"kubernetes.io/projected/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-kube-api-access-nrtgb\") pod \"payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf\" (UID: \"e377a587-5a3a-46d6-a95e-b2a0decdf9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:34:58.208319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.207478    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf\" (UID: \"e377a587-5a3a-46d6-a95e-b2a0decdf9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:34:58.388431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.388393    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkwb\" (UniqueName: \"kubernetes.io/projected/1e5b2266-6f15-44c8-b44c-e6288fae68ba-kube-api-access-djkwb\") pod \"payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh\" (UID: \"1e5b2266-6f15-44c8-b44c-e6288fae68ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:34:58.388900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.388875    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtgb\" (UniqueName: \"kubernetes.io/projected/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-kube-api-access-nrtgb\") pod \"payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf\" (UID: \"e377a587-5a3a-46d6-a95e-b2a0decdf9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:34:58.398319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.389042    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf\" (UID: \"e377a587-5a3a-46d6-a95e-b2a0decdf9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:34:58.408373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.398802    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5b2266-6f15-44c8-b44c-e6288fae68ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh\" (UID: \"1e5b2266-6f15-44c8-b44c-e6288fae68ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:34:58.428689 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.410402    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf\" (UID: \"e377a587-5a3a-46d6-a95e-b2a0decdf9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:34:58.473315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.472387    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtgb\" (UniqueName: \"kubernetes.io/projected/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-kube-api-access-nrtgb\") pod \"payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf\" (UID: \"e377a587-5a3a-46d6-a95e-b2a0decdf9c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:34:58.526320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.514820    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5b2266-6f15-44c8-b44c-e6288fae68ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh\" (UID: \"1e5b2266-6f15-44c8-b44c-e6288fae68ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:34:58.526320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.515061    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkwb\" (UniqueName: \"kubernetes.io/projected/1e5b2266-6f15-44c8-b44c-e6288fae68ba-kube-api-access-djkwb\") pod \"payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh\" (UID: \"1e5b2266-6f15-44c8-b44c-e6288fae68ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:34:58.526320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.516553    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5b2266-6f15-44c8-b44c-e6288fae68ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh\" (UID: \"1e5b2266-6f15-44c8-b44c-e6288fae68ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:34:58.577750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.577707    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkwb\" (UniqueName: \"kubernetes.io/projected/1e5b2266-6f15-44c8-b44c-e6288fae68ba-kube-api-access-djkwb\") pod \"payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh\" (UID: \"1e5b2266-6f15-44c8-b44c-e6288fae68ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:34:58.614716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.614673    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:34:58.763350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.761526    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:34:58.903317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:58.900727    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:59.378884 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:34:59.378851    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5b2266_6f15_44c8_b44c_e6288fae68ba.slice/crio-a45167b0b94cbd0a67dbb26427f066031f51b149cf8f0e2d46696283cd2ce95a WatchSource:0}: Error finding container a45167b0b94cbd0a67dbb26427f066031f51b149cf8f0e2d46696283cd2ce95a: Status 404 returned error can't find the container with id a45167b0b94cbd0a67dbb26427f066031f51b149cf8f0e2d46696283cd2ce95a
Nov 02 01:34:59.379313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:59.379277    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"]
Nov 02 01:34:59.433930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:59.433901    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"]
Nov 02 01:34:59.444675 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:34:59.441542    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode377a587_5a3a_46d6_a95e_b2a0decdf9c3.slice/crio-a7769dd7d5b654ebc42274fca456e673ab5760bb37f18d72ac35e2b03f351ade WatchSource:0}: Error finding container a7769dd7d5b654ebc42274fca456e673ab5760bb37f18d72ac35e2b03f351ade: Status 404 returned error can't find the container with id a7769dd7d5b654ebc42274fca456e673ab5760bb37f18d72ac35e2b03f351ade
Nov 02 01:34:59.761868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:59.761836    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:34:59.965642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:59.965408    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh" event={"ID":"1e5b2266-6f15-44c8-b44c-e6288fae68ba","Type":"ContainerStarted","Data":"15d44f96cd8984d2a2ca3cb9e73c13b267ca94b86cecd6507dabbff3b940136a"}
Nov 02 01:34:59.966131 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:59.965795    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh" event={"ID":"1e5b2266-6f15-44c8-b44c-e6288fae68ba","Type":"ContainerStarted","Data":"a45167b0b94cbd0a67dbb26427f066031f51b149cf8f0e2d46696283cd2ce95a"}
Nov 02 01:34:59.966492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:59.966473    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:34:59.976822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:59.976793    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf" event={"ID":"e377a587-5a3a-46d6-a95e-b2a0decdf9c3","Type":"ContainerStarted","Data":"699ed7141a3af22f0dd36e5ff6a4b268ed855c77f15e32d7f40e5e2b84802d35"}
Nov 02 01:34:59.977023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:59.977004    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf" event={"ID":"e377a587-5a3a-46d6-a95e-b2a0decdf9c3","Type":"ContainerStarted","Data":"a7769dd7d5b654ebc42274fca456e673ab5760bb37f18d72ac35e2b03f351ade"}
Nov 02 01:34:59.977176 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:34:59.977161    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:35:00.020188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:00.020083    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf" podStartSLOduration=3.0200678 podStartE2EDuration="3.0200678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:00.019403128 +0000 UTC m=+1205.324335954" watchObservedRunningTime="2026-11-02 01:35:00.0200678 +0000 UTC m=+1205.325000625"
Nov 02 01:35:00.024338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:00.024301    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh" podStartSLOduration=2.024274493 podStartE2EDuration="2.024274493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:34:59.999607083 +0000 UTC m=+1205.304539908" watchObservedRunningTime="2026-11-02 01:35:00.024274493 +0000 UTC m=+1205.329207319"
Nov 02 01:35:00.472043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:00.472003    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:00.754602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:00.754508    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:01.166784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:01.166738    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:01.405879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:01.405847    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:02.091011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:02.090977    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:03.022826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.021284    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:03.092384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.092347    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"]
Nov 02 01:35:03.185831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.185794    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"]
Nov 02 01:35:03.214703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.214212    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605ce8fc-6qbtg"]
Nov 02 01:35:03.227232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.227187    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bcb9951-gq9dc"]
Nov 02 01:35:03.239405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.237572    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"]
Nov 02 01:35:03.250277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.250247    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.255460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.255430    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:35:03.255819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.255613    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:35:03.255819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.255724    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:35:03.255819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.255794    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:35:03.257458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.257422    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"]
Nov 02 01:35:03.313031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.312951    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e438941-511f-432b-8a18-82f06a03f718-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.313182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.313137    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp64r\" (UniqueName: \"kubernetes.io/projected/8e438941-511f-432b-8a18-82f06a03f718-kube-api-access-lp64r\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.313334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.313274    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e438941-511f-432b-8a18-82f06a03f718-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.414713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.414669    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp64r\" (UniqueName: \"kubernetes.io/projected/8e438941-511f-432b-8a18-82f06a03f718-kube-api-access-lp64r\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.414948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.414824    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e438941-511f-432b-8a18-82f06a03f718-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.414948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.414925    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e438941-511f-432b-8a18-82f06a03f718-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.415575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.415549    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e438941-511f-432b-8a18-82f06a03f718-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.417385 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.417358    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e438941-511f-432b-8a18-82f06a03f718-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.427044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.427020    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp64r\" (UniqueName: \"kubernetes.io/projected/8e438941-511f-432b-8a18-82f06a03f718-kube-api-access-lp64r\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.568090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.568008    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:03.862028 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:03.861998    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"]
Nov 02 01:35:03.862522 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:35:03.862498    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e438941_511f_432b_8a18_82f06a03f718.slice/crio-bb60e83bcb364d2c3aec7b11003e84ee4459776f25c67ce71272a22454227e98 WatchSource:0}: Error finding container bb60e83bcb364d2c3aec7b11003e84ee4459776f25c67ce71272a22454227e98: Status 404 returned error can't find the container with id bb60e83bcb364d2c3aec7b11003e84ee4459776f25c67ce71272a22454227e98
Nov 02 01:35:04.050413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:04.050379    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:04.051719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:04.051693    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6" event={"ID":"8e438941-511f-432b-8a18-82f06a03f718","Type":"ContainerStarted","Data":"bb60e83bcb364d2c3aec7b11003e84ee4459776f25c67ce71272a22454227e98"}
Nov 02 01:35:04.516922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:04.516841    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:04.849757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:04.849669    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e1766f9-294d-4345-b7c8-7018f1b36581" path="/var/lib/kubelet/pods/0e1766f9-294d-4345-b7c8-7018f1b36581/volumes"
Nov 02 01:35:04.850284 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:04.850261    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc61da2-91a6-4eba-9b66-f8f375b61055" path="/var/lib/kubelet/pods/7fc61da2-91a6-4eba-9b66-f8f375b61055/volumes"
Nov 02 01:35:04.973085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:04.973056    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:05.107577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:05.107455    4851 generic.go:356] "Generic (PLEG): container finished" podID="8e438941-511f-432b-8a18-82f06a03f718" containerID="e40daa9ea8e31b04b1b1eee245187dcf52442d9f47baec929cf9e484e5d364d3" exitCode=0
Nov 02 01:35:05.108467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:05.108433    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6" event={"ID":"8e438941-511f-432b-8a18-82f06a03f718","Type":"ContainerDied","Data":"e40daa9ea8e31b04b1b1eee245187dcf52442d9f47baec929cf9e484e5d364d3"}
Nov 02 01:35:05.213096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:05.213059    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:05.976508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:05.976475    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:06.352466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.352437    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:06.377425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.377399    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:06.421994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.421965    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"]
Nov 02 01:35:06.423129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.423101    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8e438941-511f-432b-8a18-82f06a03f718" containerName="revoke-keys"
Nov 02 01:35:06.423129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.423128    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e438941-511f-432b-8a18-82f06a03f718" containerName="revoke-keys"
Nov 02 01:35:06.423405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.423380    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e438941-511f-432b-8a18-82f06a03f718" containerName="revoke-keys"
Nov 02 01:35:06.427814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.427789    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:35:06.435376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.435351    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0bbeda\""
Nov 02 01:35:06.436026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.436004    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0bbeda-dockercfg-mthwq\""
Nov 02 01:35:06.436167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.436070    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"]
Nov 02 01:35:06.446264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.446243    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:35:06.468406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.468382    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"]
Nov 02 01:35:06.478271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.478247    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"]
Nov 02 01:35:06.500577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.500552    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e438941-511f-432b-8a18-82f06a03f718-kube-api-access-lp64r\" (UniqueName: \"kubernetes.io/projected/8e438941-511f-432b-8a18-82f06a03f718-kube-api-access-lp64r\") pod \"8e438941-511f-432b-8a18-82f06a03f718\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") "
Nov 02 01:35:06.500743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.500591    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e438941-511f-432b-8a18-82f06a03f718-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e438941-511f-432b-8a18-82f06a03f718-maas-api-xks-ca\") pod \"8e438941-511f-432b-8a18-82f06a03f718\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") "
Nov 02 01:35:06.500743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.500687    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e438941-511f-432b-8a18-82f06a03f718-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e438941-511f-432b-8a18-82f06a03f718-maas-api-openshift-ca\") pod \"8e438941-511f-432b-8a18-82f06a03f718\" (UID: \"8e438941-511f-432b-8a18-82f06a03f718\") "
Nov 02 01:35:06.501056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.501024    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e438941-511f-432b-8a18-82f06a03f718-maas-api-openshift-ca" pod "8e438941-511f-432b-8a18-82f06a03f718" (UID: "8e438941-511f-432b-8a18-82f06a03f718"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:35:06.501134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.501072    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kr8d\" (UniqueName: \"kubernetes.io/projected/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-kube-api-access-2kr8d\") pod \"payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p\" (UID: \"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:35:06.501398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.501374    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p\" (UID: \"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:35:06.501738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.501715    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e438941-511f-432b-8a18-82f06a03f718-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:06.503003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.502973    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e438941-511f-432b-8a18-82f06a03f718-maas-api-xks-ca" pod "8e438941-511f-432b-8a18-82f06a03f718" (UID: "8e438941-511f-432b-8a18-82f06a03f718"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:35:06.503106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.503009    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e438941-511f-432b-8a18-82f06a03f718-kube-api-access-lp64r" pod "8e438941-511f-432b-8a18-82f06a03f718" (UID: "8e438941-511f-432b-8a18-82f06a03f718"). InnerVolumeSpecName "kube-api-access-lp64r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:35:06.605404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.603228    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kr8d\" (UniqueName: \"kubernetes.io/projected/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-kube-api-access-2kr8d\") pod \"payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p\" (UID: \"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:35:06.605404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.603438    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbn7\" (UniqueName: \"kubernetes.io/projected/ce77099f-2ea7-4032-ae62-9e089de80c22-kube-api-access-cwbn7\") pod \"payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m\" (UID: \"ce77099f-2ea7-4032-ae62-9e089de80c22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:35:06.605404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.603490    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p\" (UID: \"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:35:06.605404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.603589    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce77099f-2ea7-4032-ae62-9e089de80c22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m\" (UID: \"ce77099f-2ea7-4032-ae62-9e089de80c22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:35:06.605404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.603760    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp64r\" (UniqueName: \"kubernetes.io/projected/8e438941-511f-432b-8a18-82f06a03f718-kube-api-access-lp64r\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:06.605404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.603777    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e438941-511f-432b-8a18-82f06a03f718-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:06.605404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.604998    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p\" (UID: \"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:35:06.634957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.634921    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:06.665548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.660075    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kr8d\" (UniqueName: \"kubernetes.io/projected/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-kube-api-access-2kr8d\") pod \"payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p\" (UID: \"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:35:06.720159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.720115    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbn7\" (UniqueName: \"kubernetes.io/projected/ce77099f-2ea7-4032-ae62-9e089de80c22-kube-api-access-cwbn7\") pod \"payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m\" (UID: \"ce77099f-2ea7-4032-ae62-9e089de80c22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:35:06.720891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.720863    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce77099f-2ea7-4032-ae62-9e089de80c22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m\" (UID: \"ce77099f-2ea7-4032-ae62-9e089de80c22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:35:06.749353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.729807    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce77099f-2ea7-4032-ae62-9e089de80c22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m\" (UID: \"ce77099f-2ea7-4032-ae62-9e089de80c22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:35:06.756095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.756060    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbn7\" (UniqueName: \"kubernetes.io/projected/ce77099f-2ea7-4032-ae62-9e089de80c22-kube-api-access-cwbn7\") pod \"payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m\" (UID: \"ce77099f-2ea7-4032-ae62-9e089de80c22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:35:06.785317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.785020    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:35:06.802660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:06.802623    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:35:07.073073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:07.072070    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:07.156927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:07.156890    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6" event={"ID":"8e438941-511f-432b-8a18-82f06a03f718","Type":"ContainerDied","Data":"bb60e83bcb364d2c3aec7b11003e84ee4459776f25c67ce71272a22454227e98"}
Nov 02 01:35:07.157108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:07.156955    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb60e83bcb364d2c3aec7b11003e84ee4459776f25c67ce71272a22454227e98"
Nov 02 01:35:07.157108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:07.157032    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"
Nov 02 01:35:07.358362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:07.336360    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:07.540172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:07.540147    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"]
Nov 02 01:35:07.547512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:07.547482    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"]
Nov 02 01:35:08.079972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:08.079930    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:08.220008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:08.219966    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p" event={"ID":"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd","Type":"ContainerStarted","Data":"491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06"}
Nov 02 01:35:08.220482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:08.220457    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p" event={"ID":"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd","Type":"ContainerStarted","Data":"d40282d4e283598aaf61ecf9e6eb15a34cc891e62f0b8481f0c9dced30da1843"}
Nov 02 01:35:08.228845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:08.228808    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m" event={"ID":"ce77099f-2ea7-4032-ae62-9e089de80c22","Type":"ContainerStarted","Data":"1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9"}
Nov 02 01:35:08.229133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:08.229114    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m" event={"ID":"ce77099f-2ea7-4032-ae62-9e089de80c22","Type":"ContainerStarted","Data":"55e947ed39620baa0215435a935662f3349844303a538eb418511e4c784e20ee"}
Nov 02 01:35:08.230531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:08.230497    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:35:08.265331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:08.265261    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p" podStartSLOduration=2.26524085 podStartE2EDuration="2.26524085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:08.247353763 +0000 UTC m=+1213.552286591" watchObservedRunningTime="2026-11-02 01:35:08.26524085 +0000 UTC m=+1213.570173678"
Nov 02 01:35:08.282934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:08.282872    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m" podStartSLOduration=2.28285149 podStartE2EDuration="2.28285149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:08.279249883 +0000 UTC m=+1213.584182710" watchObservedRunningTime="2026-11-02 01:35:08.28285149 +0000 UTC m=+1213.587784317"
Nov 02 01:35:08.871327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:08.871257    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:09.237362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:09.237332    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:35:09.592609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:09.592522    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:10.389241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:10.389203    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:10.982344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:10.982315    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:35:11.089117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:11.088614    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:11.304111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:11.304015    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:11.314894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:11.314864    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"]
Nov 02 01:35:11.319560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:11.319537    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-37b78d-58pmr"]
Nov 02 01:35:11.543411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:11.543375    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:11.872279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:11.872244    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:11.991333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:11.991304    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:35:12.057395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:12.057356    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"]
Nov 02 01:35:12.065401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:12.065368    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"]
Nov 02 01:35:12.073018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:12.072989    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7f62df-hj5dr"]
Nov 02 01:35:12.078033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:12.078004    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4035642c-gk76s"]
Nov 02 01:35:12.578642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:12.578602    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:12.848957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:12.848880    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905" path="/var/lib/kubelet/pods/3ace0002-2f2b-4bbf-a5e5-8d1fb3e6a905/volumes"
Nov 02 01:35:12.849463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:12.849447    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eeabe40-8874-476e-ad67-f7e0d8f78109" path="/var/lib/kubelet/pods/5eeabe40-8874-476e-ad67-f7e0d8f78109/volumes"
Nov 02 01:35:12.849903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:12.849889    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0aff15e-598c-4f9b-955d-d183f7c7dd38" path="/var/lib/kubelet/pods/f0aff15e-598c-4f9b-955d-d183f7c7dd38/volumes"
Nov 02 01:35:13.110243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:13.110143    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:13.169771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:13.169735    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"]
Nov 02 01:35:13.328391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:13.328353    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:13.551959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:13.551818    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:13.790022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:13.789992    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:14.058644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:14.058609    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:14.303365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:14.290602    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:14.521351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:14.518851    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:15.404318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:15.400850    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:16.481372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:16.481265    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:16.692338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:16.689703    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:17.714182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:17.714147    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:17.899899 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:35:17.899844    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:35:18.178314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:18.175529    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k" podUID="6d53f56c-35b1-4fcf-be30-b503e0f45a8f" containerName="payload-pre-processing" containerID="cri-o://e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc" gracePeriod=25
Nov 02 01:35:18.732314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:18.732252    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:19.008834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:18.995833    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:19.105363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.104388    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"]
Nov 02 01:35:19.208318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.208251    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8655f22a-5l7ww"]
Nov 02 01:35:19.258314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.257880    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:19.347374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.344398    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:19.358315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.347862    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:19.405404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.403396    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:19.595564 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.595510    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:19.640672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.640535    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:19.670329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.656436    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:19.755602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.755561    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:19.782813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.782760    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:19.884196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.884163    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:19.910622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:19.910531    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:20.101261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:20.101211    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:20.134145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:20.134096    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"]
Nov 02 01:35:20.134659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:20.134627    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy" containerID="cri-o://86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3" gracePeriod=30
Nov 02 01:35:20.182421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:20.182035    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:20.295153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:20.295128    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:35:20.295384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:20.295196    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:35:20.411812 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:20.411778    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:20.966743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:20.964274    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:21.020320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.014895    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90aaf3f5-e6c3-40f9-bd39-c06c8240d012" path="/var/lib/kubelet/pods/90aaf3f5-e6c3-40f9-bd39-c06c8240d012/volumes"
Nov 02 01:35:21.171111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.171076    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:21.310072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.309988    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"]
Nov 02 01:35:21.317214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.317189    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.324892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.324860    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"]
Nov 02 01:35:21.396547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.396506    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-addad9-openshift-default-dockercfg-km8ng\""
Nov 02 01:35:21.398688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.398660    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:21.500561 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.500516    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jw2b\" (UniqueName: \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-kube-api-access-2jw2b\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.500948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.500927    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-data\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.501126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.501112    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-envoy\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.507957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.507923    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-podinfo\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.528319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.524176    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-token\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.528319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.524350    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.528319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.524431    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istiod-ca-cert\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.528319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.524570    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-certs\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.528319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.524658    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-credential-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.642369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.641532    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jw2b\" (UniqueName: \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-kube-api-access-2jw2b\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.642369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.641638    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-data\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.642369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.641674    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-envoy\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.642369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.641705    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-podinfo\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.642369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.641770    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-token\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.642369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.641880    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.642369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.641929    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istiod-ca-cert\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.642369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.642043    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-certs\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.642369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.642121    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-credential-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.642980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.642706    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-credential-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.647346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.643322    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-data\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.647346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.643833    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.662317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.655679    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-envoy\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.662317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.655968    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-certs\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.662317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.658120    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istiod-ca-cert\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.680726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.672251    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-podinfo\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.680726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.680195    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jw2b\" (UniqueName: \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-kube-api-access-2jw2b\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.721316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.720407    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-token\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:21.830317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.824720    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:22.004315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:21.991676    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:22.455880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:22.455198    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:23.164510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:23.164474    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"]
Nov 02 01:35:23.165269 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:35:23.165241    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c1d21f_cac3_4641_a6e2_92b2e12551b1.slice/crio-e5ab375b3399c57cff819dc298e52c1f68f554998fc74579a82da0102309e44f WatchSource:0}: Error finding container e5ab375b3399c57cff819dc298e52c1f68f554998fc74579a82da0102309e44f: Status 404 returned error can't find the container with id e5ab375b3399c57cff819dc298e52c1f68f554998fc74579a82da0102309e44f
Nov 02 01:35:23.168018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:23.167988    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:35:23.168083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:23.168050    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:35:23.168136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:23.168080    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:35:23.424881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:23.424800    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:24.021695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:24.021656    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" event={"ID":"b0c1d21f-cac3-4641-a6e2-92b2e12551b1","Type":"ContainerStarted","Data":"6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194"}
Nov 02 01:35:24.021873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:24.021721    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" event={"ID":"b0c1d21f-cac3-4641-a6e2-92b2e12551b1","Type":"ContainerStarted","Data":"e5ab375b3399c57cff819dc298e52c1f68f554998fc74579a82da0102309e44f"}
Nov 02 01:35:24.062654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:24.062595    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" podStartSLOduration=3.062580171 podStartE2EDuration="3.062580171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:24.059857735 +0000 UTC m=+1229.364790562" watchObservedRunningTime="2026-11-02 01:35:24.062580171 +0000 UTC m=+1229.367512996"
Nov 02 01:35:24.187068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:24.187032    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:24.871339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:24.871307    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:24.996754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:24.996668    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:25.035227 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.019675    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:25.085924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.085898    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:25.087080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.087054    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:35:25.485698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.485660    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k" podUID="6d53f56c-35b1-4fcf-be30-b503e0f45a8f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:35:25.510154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.510121    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:35:25.623537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.623496    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-envoy\") pod \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") "
Nov 02 01:35:25.623709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.623543    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-socket\") pod \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") "
Nov 02 01:35:25.623807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.623780    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-socket" pod "602e7b84-294b-482d-afb8-fb2e5a9a37ce" (UID: "602e7b84-294b-482d-afb8-fb2e5a9a37ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:35:25.623866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.623793    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istiod-ca-cert\") pod \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") "
Nov 02 01:35:25.623937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.623874    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-credential-socket\") pod \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") "
Nov 02 01:35:25.623937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.623927    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-kube-api-access-7d7xk\" (UniqueName: \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-kube-api-access-7d7xk\") pod \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") "
Nov 02 01:35:25.624044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.624021    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-data\") pod \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") "
Nov 02 01:35:25.624101 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.624069    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-podinfo\") pod \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") "
Nov 02 01:35:25.624159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.624121    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istiod-ca-cert" pod "602e7b84-294b-482d-afb8-fb2e5a9a37ce" (UID: "602e7b84-294b-482d-afb8-fb2e5a9a37ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:35:25.624222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.624151    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-credential-socket" pod "602e7b84-294b-482d-afb8-fb2e5a9a37ce" (UID: "602e7b84-294b-482d-afb8-fb2e5a9a37ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:35:25.624222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.624180    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-token\" (UniqueName: \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-token\") pod \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") "
Nov 02 01:35:25.624361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.624281    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-certs\") pod \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\" (UID: \"602e7b84-294b-482d-afb8-fb2e5a9a37ce\") "
Nov 02 01:35:25.624486 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.624438    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-data" pod "602e7b84-294b-482d-afb8-fb2e5a9a37ce" (UID: "602e7b84-294b-482d-afb8-fb2e5a9a37ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:35:25.624936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.624898    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-certs" pod "602e7b84-294b-482d-afb8-fb2e5a9a37ce" (UID: "602e7b84-294b-482d-afb8-fb2e5a9a37ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:35:25.625446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.625406    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:25.625446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.625434    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:25.625590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.625450    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:25.625590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.625467    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:25.625590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.625482    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:25.626745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.626717    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-envoy" pod "602e7b84-294b-482d-afb8-fb2e5a9a37ce" (UID: "602e7b84-294b-482d-afb8-fb2e5a9a37ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:35:25.627085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.627065    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-kube-api-access-7d7xk" pod "602e7b84-294b-482d-afb8-fb2e5a9a37ce" (UID: "602e7b84-294b-482d-afb8-fb2e5a9a37ce"). InnerVolumeSpecName "kube-api-access-7d7xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:35:25.627173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.627107    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-token" pod "602e7b84-294b-482d-afb8-fb2e5a9a37ce" (UID: "602e7b84-294b-482d-afb8-fb2e5a9a37ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:35:25.631647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.631623    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-podinfo" pod "602e7b84-294b-482d-afb8-fb2e5a9a37ce" (UID: "602e7b84-294b-482d-afb8-fb2e5a9a37ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:35:25.727206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.727106    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7xk\" (UniqueName: \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-kube-api-access-7d7xk\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:25.727206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.727146    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:25.727206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.727162    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:25.727206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.727176    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602e7b84-294b-482d-afb8-fb2e5a9a37ce-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:25.889462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:25.889421    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:26.099072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.098982    4851 generic.go:356] "Generic (PLEG): container finished" podID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerID="86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3" exitCode=0
Nov 02 01:35:26.099072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.099042    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"
Nov 02 01:35:26.099353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.099042    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" event={"ID":"602e7b84-294b-482d-afb8-fb2e5a9a37ce","Type":"ContainerDied","Data":"86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3"}
Nov 02 01:35:26.099353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.099181    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88" event={"ID":"602e7b84-294b-482d-afb8-fb2e5a9a37ce","Type":"ContainerDied","Data":"4bc9b66cbcd3da0443c7422e0b80b6b0bc5108a1800e13e77d79c6d8e11695c5"}
Nov 02 01:35:26.099353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.099207    4851 scope.go:122] "RemoveContainer" containerID="86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3"
Nov 02 01:35:26.112406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.112386    4851 scope.go:122] "RemoveContainer" containerID="29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9"
Nov 02 01:35:26.124256 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.124233    4851 scope.go:122] "RemoveContainer" containerID="86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3"
Nov 02 01:35:26.124638 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:35:26.124614    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3\": container with ID starting with 86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3 not found: ID does not exist" containerID="86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3"
Nov 02 01:35:26.124704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.124649    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3"} err="failed to get container status \"86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3\": rpc error: code = NotFound desc = could not find container \"86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3\": container with ID starting with 86e90c104dd193031bdf1dd7a930a15e17d65fe9ff7d6ab18479aac34e2befa3 not found: ID does not exist"
Nov 02 01:35:26.124704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.124667    4851 scope.go:122] "RemoveContainer" containerID="29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9"
Nov 02 01:35:26.125214 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:35:26.125170    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9\": container with ID starting with 29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9 not found: ID does not exist" containerID="29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9"
Nov 02 01:35:26.125354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.125238    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9"} err="failed to get container status \"29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9\": rpc error: code = NotFound desc = could not find container \"29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9\": container with ID starting with 29dad0fdb143b56e149316cc8df046c4b704b3499513164daa83115278ca52d9 not found: ID does not exist"
Nov 02 01:35:26.133043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.133016    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"]
Nov 02 01:35:26.137317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.137276    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-37b78d-openshift-default-65dc5b4f97-s8t88"]
Nov 02 01:35:26.467324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.467263    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"]
Nov 02 01:35:26.468507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.468480    4851 cpu_manager.go:414] "containerMap: removing container" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy"
Nov 02 01:35:26.468507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.468508    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy"
Nov 02 01:35:26.468727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.468550    4851 cpu_manager.go:414] "containerMap: removing container" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy"
Nov 02 01:35:26.468727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.468559    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy"
Nov 02 01:35:26.468838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.468744    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy"
Nov 02 01:35:26.468838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.468764    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" containerName="istio-proxy"
Nov 02 01:35:26.478371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.475081    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.479887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.479850    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"]
Nov 02 01:35:26.482559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.482510    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:35:26.486471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.486450    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:35:26.487616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.487593    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:35:26.488002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.487794    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:35:26.492318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.491255    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:26.545892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.545859    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f73a317-gnn6s\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.546216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.546190    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z48m\" (UniqueName: \"kubernetes.io/projected/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-kube-api-access-9z48m\") pod \"maas-api-revoke-sub-1f73a317-gnn6s\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.546357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.546262    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f73a317-gnn6s\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.649313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.648568    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f73a317-gnn6s\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.649313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.648752    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f73a317-gnn6s\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.649313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.649062    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z48m\" (UniqueName: \"kubernetes.io/projected/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-kube-api-access-9z48m\") pod \"maas-api-revoke-sub-1f73a317-gnn6s\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.654320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.651058    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f73a317-gnn6s\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.658324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.655911    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f73a317-gnn6s\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.733357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.732565    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z48m\" (UniqueName: \"kubernetes.io/projected/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-kube-api-access-9z48m\") pod \"maas-api-revoke-sub-1f73a317-gnn6s\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.760317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.743941    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:26.821316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.805928    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:26.942263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:26.942224    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602e7b84-294b-482d-afb8-fb2e5a9a37ce" path="/var/lib/kubelet/pods/602e7b84-294b-482d-afb8-fb2e5a9a37ce/volumes"
Nov 02 01:35:27.017365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.001223    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:27.244860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.244815    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:27.695795 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.695174    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"]
Nov 02 01:35:27.750524 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:35:27.750486    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67176ea1_a6c8_4bbb_9cfc_457f12f1c645.slice/crio-2b0c31be30ea26dd168f9dfa00420e92004b8b6f69b35f087c91700b255ad52d WatchSource:0}: Error finding container 2b0c31be30ea26dd168f9dfa00420e92004b8b6f69b35f087c91700b255ad52d: Status 404 returned error can't find the container with id 2b0c31be30ea26dd168f9dfa00420e92004b8b6f69b35f087c91700b255ad52d
Nov 02 01:35:27.837636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.825840    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"]
Nov 02 01:35:27.874183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.872864    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"]
Nov 02 01:35:27.874183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.873025    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:27.907099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.906117    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-00f1da-openshift-default-dockercfg-c5svk\""
Nov 02 01:35:27.909485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.907738    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:27.929127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.928627    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:27.929127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.928712    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:27.929127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.928761    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:27.929127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.928825    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da59dac5-1239-4ee9-b7af-f949e9891687-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:27.931608 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.928909    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da59dac5-1239-4ee9-b7af-f949e9891687-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:27.932875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.931904    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:27.932875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.931998    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:27.932875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.932067    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6pck\" (UniqueName: \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-kube-api-access-s6pck\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:27.932875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:27.932271    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.039447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.039412    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.045615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.045582    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.045920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.045895    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6pck\" (UniqueName: \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-kube-api-access-s6pck\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.046266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.046247    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.046524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.046503    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.046711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.046693    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.063739 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.063699    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.064036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.064018    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da59dac5-1239-4ee9-b7af-f949e9891687-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.064244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.064226    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da59dac5-1239-4ee9-b7af-f949e9891687-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.069327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.040143    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.069327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.056864    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.069327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.057094    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.069327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.057853    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.077671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.077581    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da59dac5-1239-4ee9-b7af-f949e9891687-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.092121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.092086    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6pck\" (UniqueName: \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-kube-api-access-s6pck\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.093256 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.093207    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.125694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.125608    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:28.127341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.127318    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.173947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.169621    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da59dac5-1239-4ee9-b7af-f949e9891687-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.186253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.186208    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s" event={"ID":"67176ea1-a6c8-4bbb-9cfc-457f12f1c645","Type":"ContainerStarted","Data":"2b0c31be30ea26dd168f9dfa00420e92004b8b6f69b35f087c91700b255ad52d"}
Nov 02 01:35:28.408088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.408017    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:28.440064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.439802    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:28.506952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.506918    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"]
Nov 02 01:35:28.526126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.526085    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:35:28.603316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.568204    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"]
Nov 02 01:35:28.603316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.583950    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-addad9\""
Nov 02 01:35:28.603316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.584087    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-addad9-dockercfg-qcdf5\""
Nov 02 01:35:28.667911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.628612    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a157216-0fe8-496b-811e-35ee545e3e31-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s\" (UID: \"9a157216-0fe8-496b-811e-35ee545e3e31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:35:28.667911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.628720    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxf8\" (UniqueName: \"kubernetes.io/projected/9a157216-0fe8-496b-811e-35ee545e3e31-kube-api-access-cvxf8\") pod \"payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s\" (UID: \"9a157216-0fe8-496b-811e-35ee545e3e31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:35:28.667911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.653384    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:28.821870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.731523    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a157216-0fe8-496b-811e-35ee545e3e31-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s\" (UID: \"9a157216-0fe8-496b-811e-35ee545e3e31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:35:28.821870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.731615    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxf8\" (UniqueName: \"kubernetes.io/projected/9a157216-0fe8-496b-811e-35ee545e3e31-kube-api-access-cvxf8\") pod \"payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s\" (UID: \"9a157216-0fe8-496b-811e-35ee545e3e31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:35:28.821870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.733089    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a157216-0fe8-496b-811e-35ee545e3e31-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s\" (UID: \"9a157216-0fe8-496b-811e-35ee545e3e31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:35:28.863039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.859537    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxf8\" (UniqueName: \"kubernetes.io/projected/9a157216-0fe8-496b-811e-35ee545e3e31-kube-api-access-cvxf8\") pod \"payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s\" (UID: \"9a157216-0fe8-496b-811e-35ee545e3e31\") " pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:35:28.918641 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:28.913478    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:35:29.062216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.062163    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"]
Nov 02 01:35:29.101254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.101208    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:35:29.101396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.101333    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:35:29.102316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.101498    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:35:29.203437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.203402    4851 generic.go:356] "Generic (PLEG): container finished" podID="67176ea1-a6c8-4bbb-9cfc-457f12f1c645" containerID="9b9c2cfa9f7c8bbd1003c70d1479dc7cc6daccc9cbbae853a96ecc8ac0822ad3" exitCode=0
Nov 02 01:35:29.203614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.203518    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s" event={"ID":"67176ea1-a6c8-4bbb-9cfc-457f12f1c645","Type":"ContainerDied","Data":"9b9c2cfa9f7c8bbd1003c70d1479dc7cc6daccc9cbbae853a96ecc8ac0822ad3"}
Nov 02 01:35:29.206751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.205839    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" event={"ID":"da59dac5-1239-4ee9-b7af-f949e9891687","Type":"ContainerStarted","Data":"16cd0a698ded695787a2270dfa852a51ca3d982e39533d5f49423977246062a5"}
Nov 02 01:35:29.280485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.280451    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:29.497948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.497095    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:29.507432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.507316    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"]
Nov 02 01:35:29.978059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:29.978023    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:30.026351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.026257    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"]
Nov 02 01:35:30.138659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.136612    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aba21e8-6z8sj"]
Nov 02 01:35:30.218148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.218114    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" event={"ID":"da59dac5-1239-4ee9-b7af-f949e9891687","Type":"ContainerStarted","Data":"b9a724f46926be34a1037c594310b576b54f2ab11d96c0bb0170a59bb440c0fb"}
Nov 02 01:35:30.220086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.220058    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s" event={"ID":"9a157216-0fe8-496b-811e-35ee545e3e31","Type":"ContainerStarted","Data":"c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376"}
Nov 02 01:35:30.220215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.220113    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s" event={"ID":"9a157216-0fe8-496b-811e-35ee545e3e31","Type":"ContainerStarted","Data":"920a491edf667a0ae7c5e270fff2dfbbebc6c3f2e95c58427f85d24b302beb2e"}
Nov 02 01:35:30.220215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.220206    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:35:30.246666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.246554    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podStartSLOduration=3.246539393 podStartE2EDuration="3.246539393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:30.241740704 +0000 UTC m=+1235.546673533" watchObservedRunningTime="2026-11-02 01:35:30.246539393 +0000 UTC m=+1235.551472216"
Nov 02 01:35:30.267387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.267331    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s" podStartSLOduration=2.267319972 podStartE2EDuration="2.267319972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:30.263460814 +0000 UTC m=+1235.568393640" watchObservedRunningTime="2026-11-02 01:35:30.267319972 +0000 UTC m=+1235.572252799"
Nov 02 01:35:30.455332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.447367    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:30.514549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.474789    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:30.580387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.580272    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:30.696323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.692018    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-kube-api-access-9z48m\" (UniqueName: \"kubernetes.io/projected/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-kube-api-access-9z48m\") pod \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") "
Nov 02 01:35:30.696323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.692190    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-xks-ca\") pod \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") "
Nov 02 01:35:30.782552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.782471    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-openshift-ca\") pod \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\" (UID: \"67176ea1-a6c8-4bbb-9cfc-457f12f1c645\") "
Nov 02 01:35:30.995897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.801503    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-openshift-ca" pod "67176ea1-a6c8-4bbb-9cfc-457f12f1c645" (UID: "67176ea1-a6c8-4bbb-9cfc-457f12f1c645"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:35:30.995897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.938847    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-xks-ca" pod "67176ea1-a6c8-4bbb-9cfc-457f12f1c645" (UID: "67176ea1-a6c8-4bbb-9cfc-457f12f1c645"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:35:30.995897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:30.944416    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-kube-api-access-9z48m" pod "67176ea1-a6c8-4bbb-9cfc-457f12f1c645" (UID: "67176ea1-a6c8-4bbb-9cfc-457f12f1c645"). InnerVolumeSpecName "kube-api-access-9z48m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:35:31.031457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.031053    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209f7830-55fc-48b9-8347-5bd2c24ce5e2" path="/var/lib/kubelet/pods/209f7830-55fc-48b9-8347-5bd2c24ce5e2/volumes"
Nov 02 01:35:31.051698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.051649    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:31.097354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.097322    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z48m\" (UniqueName: \"kubernetes.io/projected/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-kube-api-access-9z48m\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:31.097354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.097353    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:31.097550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.097363    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67176ea1-a6c8-4bbb-9cfc-457f12f1c645-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:31.301655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.300412    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:31.349615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.349581    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"
Nov 02 01:35:31.358376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.350403    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s" event={"ID":"67176ea1-a6c8-4bbb-9cfc-457f12f1c645","Type":"ContainerDied","Data":"2b0c31be30ea26dd168f9dfa00420e92004b8b6f69b35f087c91700b255ad52d"}
Nov 02 01:35:31.358376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.350450    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b0c31be30ea26dd168f9dfa00420e92004b8b6f69b35f087c91700b255ad52d"
Nov 02 01:35:31.358376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.350471    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:31.358376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.352809    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:35:31.452070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.448950    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:35:31.452070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:31.449011    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:32.244376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:32.244344    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:32.573885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:32.573200    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:32.786587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:32.786537    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:33.307973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.307942    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:33.470775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.470724    4851 generic.go:356] "Generic (PLEG): container finished" podID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerID="cc3967322c448e22c1ec6f9ce1284315850dfe653741ac4d0aa355da5cbc713b" exitCode=0
Nov 02 01:35:33.476991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.476746    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" event={"ID":"5e310f6b-1d42-4471-a2a2-3562f8199803","Type":"ContainerDied","Data":"cc3967322c448e22c1ec6f9ce1284315850dfe653741ac4d0aa355da5cbc713b"}
Nov 02 01:35:33.477825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.477760    4851 scope.go:122] "RemoveContainer" containerID="cc3967322c448e22c1ec6f9ce1284315850dfe653741ac4d0aa355da5cbc713b"
Nov 02 01:35:33.488892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.488840    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:35:33.489051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.489015    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:35:33.489129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.489118    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:35:33.797384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.797339    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:33.859218 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.859119    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"]
Nov 02 01:35:33.859912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.859887    4851 cpu_manager.go:414] "containerMap: removing container" podUID="67176ea1-a6c8-4bbb-9cfc-457f12f1c645" containerName="revoke-keys"
Nov 02 01:35:33.859912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.859906    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67176ea1-a6c8-4bbb-9cfc-457f12f1c645" containerName="revoke-keys"
Nov 02 01:35:33.860089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.860056    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="67176ea1-a6c8-4bbb-9cfc-457f12f1c645" containerName="revoke-keys"
Nov 02 01:35:33.863884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.863867    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:35:33.868861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.868836    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-00f1da\""
Nov 02 01:35:33.876188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.876167    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"]
Nov 02 01:35:33.880993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.880926    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-00f1da-dockercfg-q6rk9\""
Nov 02 01:35:33.979317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.968956    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm98p\" (UniqueName: \"kubernetes.io/projected/bb9669fc-e378-4a4b-aea5-ea97c9b71882-kube-api-access-fm98p\") pod \"payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh\" (UID: \"bb9669fc-e378-4a4b-aea5-ea97c9b71882\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:35:33.979317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:33.969179    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9669fc-e378-4a4b-aea5-ea97c9b71882-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh\" (UID: \"bb9669fc-e378-4a4b-aea5-ea97c9b71882\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:35:34.129117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.100607    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:34.141313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.131649    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm98p\" (UniqueName: \"kubernetes.io/projected/bb9669fc-e378-4a4b-aea5-ea97c9b71882-kube-api-access-fm98p\") pod \"payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh\" (UID: \"bb9669fc-e378-4a4b-aea5-ea97c9b71882\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:35:34.141313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.131832    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9669fc-e378-4a4b-aea5-ea97c9b71882-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh\" (UID: \"bb9669fc-e378-4a4b-aea5-ea97c9b71882\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:35:34.141313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.132936    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9669fc-e378-4a4b-aea5-ea97c9b71882-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh\" (UID: \"bb9669fc-e378-4a4b-aea5-ea97c9b71882\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:35:34.293768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.291596    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm98p\" (UniqueName: \"kubernetes.io/projected/bb9669fc-e378-4a4b-aea5-ea97c9b71882-kube-api-access-fm98p\") pod \"payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh\" (UID: \"bb9669fc-e378-4a4b-aea5-ea97c9b71882\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:35:34.492449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.489923    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:35:34.496630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.496599    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" event={"ID":"5e310f6b-1d42-4471-a2a2-3562f8199803","Type":"ContainerStarted","Data":"0d0888dfa1c0c908d9717a0f95aebdb8a7a14bccebd487e7e15648342c542141"}
Nov 02 01:35:34.499750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.499722    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:34.876625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.876595    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:35:34.876801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.876668    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 01:35:34.876801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.876728    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 01:35:34.964601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:34.964569    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"]
Nov 02 01:35:34.969439 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:35:34.969369    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9669fc_e378_4a4b_aea5_ea97c9b71882.slice/crio-d41e89e3d876899d46f195221169f73f101e23ab6a0f6e8d784a02cbc0c52239 WatchSource:0}: Error finding container d41e89e3d876899d46f195221169f73f101e23ab6a0f6e8d784a02cbc0c52239: Status 404 returned error can't find the container with id d41e89e3d876899d46f195221169f73f101e23ab6a0f6e8d784a02cbc0c52239
Nov 02 01:35:35.178638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:35.178603    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:35.475369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:35.470645    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:35.522935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:35.520658    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k" podUID="6d53f56c-35b1-4fcf-be30-b503e0f45a8f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:35:35.545344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:35.544425    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh" event={"ID":"bb9669fc-e378-4a4b-aea5-ea97c9b71882","Type":"ContainerStarted","Data":"f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be"}
Nov 02 01:35:35.545344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:35.544488    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh" event={"ID":"bb9669fc-e378-4a4b-aea5-ea97c9b71882","Type":"ContainerStarted","Data":"d41e89e3d876899d46f195221169f73f101e23ab6a0f6e8d784a02cbc0c52239"}
Nov 02 01:35:35.545344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:35.544535    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:35:35.588315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:35.571517    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh" podStartSLOduration=2.571497172 podStartE2EDuration="2.571497172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:35.570750324 +0000 UTC m=+1240.875683167" watchObservedRunningTime="2026-11-02 01:35:35.571497172 +0000 UTC m=+1240.876430000"
Nov 02 01:35:35.889475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:35.888427    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 01:35:35.889475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:35.888476    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 01:35:35.956666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:35.956624    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:36.200674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:36.200638    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:36.521090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:36.521002    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:36.768977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:36.768911    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:36.883386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:36.878840    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 01:35:36.883386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:36.878914    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 01:35:37.242374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:37.242336    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:37.877347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:37.877280    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 01:35:37.877753 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:37.877380    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 01:35:37.987856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:37.987816    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:38.482692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.482671    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:35:38.573143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.573106    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-kube-api-access-cz2w5\" (UniqueName: \"kubernetes.io/projected/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-kube-api-access-cz2w5\") pod \"6d53f56c-35b1-4fcf-be30-b503e0f45a8f\" (UID: \"6d53f56c-35b1-4fcf-be30-b503e0f45a8f\") "
Nov 02 01:35:38.573446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.573424    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-plugins-config-volume\") pod \"6d53f56c-35b1-4fcf-be30-b503e0f45a8f\" (UID: \"6d53f56c-35b1-4fcf-be30-b503e0f45a8f\") "
Nov 02 01:35:38.574399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.573934    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-plugins-config-volume" pod "6d53f56c-35b1-4fcf-be30-b503e0f45a8f" (UID: "6d53f56c-35b1-4fcf-be30-b503e0f45a8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:35:38.574399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.574366    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:38.576375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.576350    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-kube-api-access-cz2w5" pod "6d53f56c-35b1-4fcf-be30-b503e0f45a8f" (UID: "6d53f56c-35b1-4fcf-be30-b503e0f45a8f"). InnerVolumeSpecName "kube-api-access-cz2w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:35:38.676217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.676181    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2w5\" (UniqueName: \"kubernetes.io/projected/6d53f56c-35b1-4fcf-be30-b503e0f45a8f-kube-api-access-cz2w5\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:35:38.693084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.693047    4851 generic.go:356] "Generic (PLEG): container finished" podID="6d53f56c-35b1-4fcf-be30-b503e0f45a8f" containerID="e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc" exitCode=0
Nov 02 01:35:38.693257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.693092    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"
Nov 02 01:35:38.693257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.693139    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k" event={"ID":"6d53f56c-35b1-4fcf-be30-b503e0f45a8f","Type":"ContainerDied","Data":"e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc"}
Nov 02 01:35:38.693257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.693193    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k" event={"ID":"6d53f56c-35b1-4fcf-be30-b503e0f45a8f","Type":"ContainerDied","Data":"d19bc4c46ac0a48d17760487e70f4e235df26ce1ea121966ed4e4cea8b8c4797"}
Nov 02 01:35:38.693257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.693222    4851 scope.go:122] "RemoveContainer" containerID="e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc"
Nov 02 01:35:38.705802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.705780    4851 scope.go:122] "RemoveContainer" containerID="e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc"
Nov 02 01:35:38.706105 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:35:38.706082    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc\": container with ID starting with e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc not found: ID does not exist" containerID="e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc"
Nov 02 01:35:38.706202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.706111    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc"} err="failed to get container status \"e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc\": rpc error: code = NotFound desc = could not find container \"e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc\": container with ID starting with e3543ad5e73b38e7cdf5c71a24cbe766de9891aedd1ba13444d5aceed46d41cc not found: ID does not exist"
Nov 02 01:35:38.720778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.720758    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"]
Nov 02 01:35:38.725824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.725803    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-5mq7k"]
Nov 02 01:35:38.874600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.874564    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d53f56c-35b1-4fcf-be30-b503e0f45a8f" path="/var/lib/kubelet/pods/6d53f56c-35b1-4fcf-be30-b503e0f45a8f/volumes"
Nov 02 01:35:38.876707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.876676    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 01:35:38.876845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:38.876745    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 01:35:39.251316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:39.232596    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:39.757317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:39.754747    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:39.881598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:39.881428    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 01:35:39.881598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:39.881487    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 01:35:39.980187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:39.980153    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:40.349140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:40.349111    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:40.571251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:40.571209    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:40.915315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:40.910615    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:35:40.915315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:40.912407    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:40.915315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:40.912964    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:35:40.915315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:40.913934    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:35:40.993095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:40.993066    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"]
Nov 02 01:35:40.994079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:40.994057    4851 cpu_manager.go:414] "containerMap: removing container" podUID="6d53f56c-35b1-4fcf-be30-b503e0f45a8f" containerName="payload-pre-processing"
Nov 02 01:35:40.994183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:40.994173    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d53f56c-35b1-4fcf-be30-b503e0f45a8f" containerName="payload-pre-processing"
Nov 02 01:35:40.994444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:40.994432    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d53f56c-35b1-4fcf-be30-b503e0f45a8f" containerName="payload-pre-processing"
Nov 02 01:35:41.002549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.002526    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:35:41.006662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.005722    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"]
Nov 02 01:35:41.008331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.008275    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ffe5c1\""
Nov 02 01:35:41.008597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.008585    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ffe5c1-dockercfg-xn2ws\""
Nov 02 01:35:41.040832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.040651    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtnz\" (UniqueName: \"kubernetes.io/projected/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-kube-api-access-vrtnz\") pod \"payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns\" (UID: \"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:35:41.055344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.041073    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns\" (UID: \"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:35:41.124998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.124959    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:41.144079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.143920    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtnz\" (UniqueName: \"kubernetes.io/projected/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-kube-api-access-vrtnz\") pod \"payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns\" (UID: \"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:35:41.145727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.145698    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns\" (UID: \"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:35:41.147008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.146985    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns\" (UID: \"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:35:41.226364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.223984    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtnz\" (UniqueName: \"kubernetes.io/projected/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-kube-api-access-vrtnz\") pod \"payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns\" (UID: \"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:35:41.470353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.466077    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:35:41.521496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.521402    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:35:41.848838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:41.848807    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:42.303189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:42.303156    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"]
Nov 02 01:35:42.304273 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:35:42.304245    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4084a05c_f8e7_4e06_9aa0_1e7e8f400dd8.slice/crio-09681f92c31952c3e575197b4da8c33896352a4bc9382989289c1e4f6f649cae WatchSource:0}: Error finding container 09681f92c31952c3e575197b4da8c33896352a4bc9382989289c1e4f6f649cae: Status 404 returned error can't find the container with id 09681f92c31952c3e575197b4da8c33896352a4bc9382989289c1e4f6f649cae
Nov 02 01:35:42.551077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:42.550994    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:42.737818 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:42.737775    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"]
Nov 02 01:35:42.745944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:42.745897    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-nnxz6"]
Nov 02 01:35:42.763689 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:42.763658    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:43.021836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:43.009603    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns" podStartSLOduration=3.009584113 podStartE2EDuration="3.009584113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:43.009115141 +0000 UTC m=+1248.314047980" watchObservedRunningTime="2026-11-02 01:35:43.009584113 +0000 UTC m=+1248.314516940"
Nov 02 01:35:43.037339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:43.037278    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:43.087011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:43.073730    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e438941-511f-432b-8a18-82f06a03f718" path="/var/lib/kubelet/pods/8e438941-511f-432b-8a18-82f06a03f718/volumes"
Nov 02 01:35:43.087011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:43.074258    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:35:43.087011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:43.074381    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns" event={"ID":"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8","Type":"ContainerStarted","Data":"0be195cf93747fd247b588c8c82c5ba4bbccdcefe86e3128c35589133c548653"}
Nov 02 01:35:43.087011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:43.074412    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns" event={"ID":"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8","Type":"ContainerStarted","Data":"09681f92c31952c3e575197b4da8c33896352a4bc9382989289c1e4f6f649cae"}
Nov 02 01:35:44.084205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:44.084172    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:44.782672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:44.782640    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:45.496659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:45.496623    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:46.231372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:46.231333    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:46.917931 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:46.917894    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:47.141803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:47.141765    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:47.375746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:47.375640    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:47.658230 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:35:47.658148    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:35:47.667537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:47.667507    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:47.685393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:47.685370    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:35:48.315883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:48.315855    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:48.804587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:48.804557    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:49.206860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:49.206825    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:50.077069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:50.077038    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:50.290004 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:50.289963    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:50.514520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:50.514489    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:50.776378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:50.770137    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:51.036037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.035904    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:51.517317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.516921    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"]
Nov 02 01:35:51.676003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.675845    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"]
Nov 02 01:35:51.676364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.676144    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:35:51.680615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.680590    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-50537f4f-dockercfg-gx2q4\""
Nov 02 01:35:51.681922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.681904    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-50537f4f\""
Nov 02 01:35:51.722598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.722569    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c4b872-5d76-4900-b4ea-09331641a617-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk\" (UID: \"10c4b872-5d76-4900-b4ea-09331641a617\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:35:51.722863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.722842    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knfkz\" (UniqueName: \"kubernetes.io/projected/10c4b872-5d76-4900-b4ea-09331641a617-kube-api-access-knfkz\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk\" (UID: \"10c4b872-5d76-4900-b4ea-09331641a617\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:35:51.824235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.824148    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knfkz\" (UniqueName: \"kubernetes.io/projected/10c4b872-5d76-4900-b4ea-09331641a617-kube-api-access-knfkz\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk\" (UID: \"10c4b872-5d76-4900-b4ea-09331641a617\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:35:51.824235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.824225    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c4b872-5d76-4900-b4ea-09331641a617-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk\" (UID: \"10c4b872-5d76-4900-b4ea-09331641a617\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:35:51.824928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.824908    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c4b872-5d76-4900-b4ea-09331641a617-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk\" (UID: \"10c4b872-5d76-4900-b4ea-09331641a617\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:35:51.839367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.839342    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knfkz\" (UniqueName: \"kubernetes.io/projected/10c4b872-5d76-4900-b4ea-09331641a617-kube-api-access-knfkz\") pod \"payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk\" (UID: \"10c4b872-5d76-4900-b4ea-09331641a617\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:35:51.865175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:51.865152    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:52.006315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:52.005597    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:35:52.353976 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:35:52.353937    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c4b872_5d76_4900_b4ea_09331641a617.slice/crio-543fa9c3264df321f19c177ffb8b6ebdad137ab7ec05c93560e5aa3505947b26 WatchSource:0}: Error finding container 543fa9c3264df321f19c177ffb8b6ebdad137ab7ec05c93560e5aa3505947b26: Status 404 returned error can't find the container with id 543fa9c3264df321f19c177ffb8b6ebdad137ab7ec05c93560e5aa3505947b26
Nov 02 01:35:52.356922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:52.356894    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"]
Nov 02 01:35:52.589390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:52.589353    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:52.668318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:52.665002    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk" event={"ID":"10c4b872-5d76-4900-b4ea-09331641a617","Type":"ContainerStarted","Data":"3a5fdf2bd6a64649919a647a1f3d18c7b9bd7ae7e508581d21a8e39de92ee9a7"}
Nov 02 01:35:52.668318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:52.665081    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk" event={"ID":"10c4b872-5d76-4900-b4ea-09331641a617","Type":"ContainerStarted","Data":"543fa9c3264df321f19c177ffb8b6ebdad137ab7ec05c93560e5aa3505947b26"}
Nov 02 01:35:52.668318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:52.665774    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:35:52.745518 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:52.745453    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk" podStartSLOduration=1.7454328669999999 podStartE2EDuration="1.745432867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:52.713597976 +0000 UTC m=+1258.018530803" watchObservedRunningTime="2026-11-02 01:35:52.745432867 +0000 UTC m=+1258.050365694"
Nov 02 01:35:53.473802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:53.473766    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:53.975275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:53.975246    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:35:54.269438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:54.269360    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:55.278654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:55.278625    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:55.512367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:55.512334    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:55.728267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:55.728228    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:56.028029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.027940    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:56.196319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.193465    4851 scope.go:122] "RemoveContainer" containerID="b598f8970b2af86572800e22efc3db592b9901b84813b6f585bba6d6f5c1ee52"
Nov 02 01:35:56.295553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.293144    4851 scope.go:122] "RemoveContainer" containerID="77b848b15aec3569b988502ab8ee29aea247569c6415e70fd8d3d55046c1407a"
Nov 02 01:35:56.312584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.312559    4851 scope.go:122] "RemoveContainer" containerID="394daf5ab7d8b74bc3776e755dc130c7751b037aa6322590dfedaa504d3fc74f"
Nov 02 01:35:56.326350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.326210    4851 scope.go:122] "RemoveContainer" containerID="f529cc7c5c46efb25fa8b794ace2dded2b7b3b14a75043030d7974619f42f04d"
Nov 02 01:35:56.339371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.339349    4851 scope.go:122] "RemoveContainer" containerID="0f1f9eeed062df6b11ae7de9a4a2dca1e0ffb46f2ad38528a6d13df5f7d1dd68"
Nov 02 01:35:56.384837 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.384804    4851 scope.go:122] "RemoveContainer" containerID="65ebcc233b36327449236c5eebf39a66c7506d1f6c0786f5f800f1290b0d8af4"
Nov 02 01:35:56.411527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.411502    4851 scope.go:122] "RemoveContainer" containerID="9da58647699082fd7610ac4abe6b56284479ffae3551ec96f224ca9b6abfd8b2"
Nov 02 01:35:56.423457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.423430    4851 scope.go:122] "RemoveContainer" containerID="f312c298ac3a54e7c1006b2ba8732b7b24ea76285681f620a84f6aac29ad4999"
Nov 02 01:35:56.435342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.435318    4851 scope.go:122] "RemoveContainer" containerID="9117e5f045bfe09ce52ac871e2ccf47f7d069e80079e894da4965f4f33893069"
Nov 02 01:35:56.446905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.446876    4851 scope.go:122] "RemoveContainer" containerID="b857e1148f9e0dd33ebbead7ab4097ea06e25e7bb208137536725cbf105dc0e4"
Nov 02 01:35:56.458317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.458280    4851 scope.go:122] "RemoveContainer" containerID="c5b7d4b86d78db9d52922e0c97d64852c75b2e56859fb04524b2672dc1693644"
Nov 02 01:35:56.470430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.470400    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:56.473167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.473147    4851 scope.go:122] "RemoveContainer" containerID="959be49370e30c729c9c9b151c6ce9d2929db3e927f19f8b348b65bacb0373b8"
Nov 02 01:35:56.488043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.488016    4851 scope.go:122] "RemoveContainer" containerID="86895bb91f8d1c6e40e33afa1f35e3329624fa095e9bfcaa00fdf6102f789b9b"
Nov 02 01:35:56.500111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.500091    4851 scope.go:122] "RemoveContainer" containerID="7b09c184dea8f11ce1fc4fd5697202ca2d31e0196b05b9a31475c39b8ed988e2"
Nov 02 01:35:56.514396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.514372    4851 scope.go:122] "RemoveContainer" containerID="48e14f9e0a78cc09e1271bbb6ee21102d44108d9d48e5a6ddc2bf54fbc32e7a6"
Nov 02 01:35:56.642627 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.634715    4851 scope.go:122] "RemoveContainer" containerID="36e168260dd6b80f8f1597aa2ea44b67ef28f371972afe46b3e46329b76cc229"
Nov 02 01:35:56.696619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.696597    4851 scope.go:122] "RemoveContainer" containerID="701a651c2ff0db01aa7e49d5e4e9ea83c27330a5086f204a883c1f4fdfab43b7"
Nov 02 01:35:56.709450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.709428    4851 scope.go:122] "RemoveContainer" containerID="d09c8252604f9385343e29e9420a8d2d4f636527e331d9752ab6a5c3f7759db0"
Nov 02 01:35:56.724120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.724093    4851 scope.go:122] "RemoveContainer" containerID="9b47e6449d6a58080d921547dcc5fc4de1bceeb5c8c44cd7e39e24e7e45e87e0"
Nov 02 01:35:56.752914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.752886    4851 scope.go:122] "RemoveContainer" containerID="6e1869f8ac8193dab9a2beb9230d9bcfddb08401e2cb93ed53f974f348baa064"
Nov 02 01:35:56.852353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:56.851843    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:57.076585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:57.076540    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:57.872015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:57.871605    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:59.259572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:59.259184    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:35:59.593371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:59.592392    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:35:59.631440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:35:59.631392    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:00.431818 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:00.431779    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:00.650772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:00.650740    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:01.735639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:01.735604    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:02.256318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:02.251452    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:02.609355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:02.609260    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:03.004346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.001649    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"]
Nov 02 01:36:03.015991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.010439    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.032188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.031858    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"]
Nov 02 01:36:03.069126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.069064    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:03.105471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.105447    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2589b2-openshift-default-dockercfg-mt8kv\""
Nov 02 01:36:03.226328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.225848    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-envoy\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.226328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.225936    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istiod-ca-cert\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.226328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.226003    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-certs\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.263367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.260418    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.263367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.260515    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-token\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.263367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.260630    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-credential-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.263367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.260700    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-data\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.263367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.260899    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zw87\" (UniqueName: \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-kube-api-access-4zw87\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.263367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.260933    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-podinfo\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.286339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.284818    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.370262    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-certs\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.370686    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-certs\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.370919    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.371206    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.371261    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-token\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.371678    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-credential-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.371943    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-credential-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.372016    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-data\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.372239    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-data\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.372458    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zw87\" (UniqueName: \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-kube-api-access-4zw87\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.372489    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-podinfo\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.373268    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-envoy\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.383364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.373809    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istiod-ca-cert\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.385022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.384951    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istiod-ca-cert\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.465313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.444457    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-token\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.465313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.444765    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-envoy\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.465313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.447724    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-podinfo\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.501346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.499919    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zw87\" (UniqueName: \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-kube-api-access-4zw87\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.684080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.683869    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:03.689616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:03.689590    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:36:04.268040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:04.267347    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"]
Nov 02 01:36:04.271790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:04.271755    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:04.271873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:04.271838    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:04.271919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:04.271880    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:04.575956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:04.575792    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:04.999718 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:04.999684    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" event={"ID":"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa","Type":"ContainerStarted","Data":"2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41"}
Nov 02 01:36:04.999936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:04.999734    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" event={"ID":"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa","Type":"ContainerStarted","Data":"69ce08fb9345414c5846d711e5700f6e106e385d1cc6e6df16e19014a24425a2"}
Nov 02 01:36:05.027178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:05.027121    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" podStartSLOduration=3.027105947 podStartE2EDuration="3.027105947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:05.023430118 +0000 UTC m=+1270.328362940" watchObservedRunningTime="2026-11-02 01:36:05.027105947 +0000 UTC m=+1270.332038772"
Nov 02 01:36:05.687464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:05.687421    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:05.693816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:05.693782    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:05.735800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:05.735763    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:06.011450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:06.011373    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:06.012585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:06.012560    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:36:06.825642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:06.825596    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:07.334133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:07.334094    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:07.554412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:07.554373    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:07.881212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:07.881177    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:08.287553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:08.287511    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:08.681405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:08.681368    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:08.751260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:08.751222    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"]
Nov 02 01:36:08.840788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:08.840758    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:36:08.850601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:08.850578    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2589b2\""
Nov 02 01:36:08.850875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:08.850859    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2589b2-dockercfg-z8rc6\""
Nov 02 01:36:08.852131 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:08.852114    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"]
Nov 02 01:36:08.891432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:08.891341    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:09.000382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:08.985102    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28hpp\" (UniqueName: \"kubernetes.io/projected/21310443-4ec4-40ae-91da-65292ae655f9-kube-api-access-28hpp\") pod \"payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4\" (UID: \"21310443-4ec4-40ae-91da-65292ae655f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:36:09.000382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:08.985231    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21310443-4ec4-40ae-91da-65292ae655f9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4\" (UID: \"21310443-4ec4-40ae-91da-65292ae655f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:36:09.089804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:09.088164    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28hpp\" (UniqueName: \"kubernetes.io/projected/21310443-4ec4-40ae-91da-65292ae655f9-kube-api-access-28hpp\") pod \"payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4\" (UID: \"21310443-4ec4-40ae-91da-65292ae655f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:36:09.089804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:09.088303    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21310443-4ec4-40ae-91da-65292ae655f9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4\" (UID: \"21310443-4ec4-40ae-91da-65292ae655f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:36:09.089804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:09.089392    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21310443-4ec4-40ae-91da-65292ae655f9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4\" (UID: \"21310443-4ec4-40ae-91da-65292ae655f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:36:09.116129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:09.116051    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28hpp\" (UniqueName: \"kubernetes.io/projected/21310443-4ec4-40ae-91da-65292ae655f9-kube-api-access-28hpp\") pod \"payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4\" (UID: \"21310443-4ec4-40ae-91da-65292ae655f9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:36:09.210625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:09.210583    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:36:09.400609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:09.400520    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:09.745837 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:09.745803    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"]
Nov 02 01:36:09.746250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:09.746093    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy" containerID="cri-o://e6e832bda8cd39fac1bb903fff56a2ce4275d404ae25c966249d8b4297db960b" gracePeriod=30
Nov 02 01:36:09.891443 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:09.891400    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21310443_4ec4_40ae_91da_65292ae655f9.slice/crio-42e130fca4aa7af6dd9085eb24d9355f16d234aee78e4660fa55be6f2ceb9d49 WatchSource:0}: Error finding container 42e130fca4aa7af6dd9085eb24d9355f16d234aee78e4660fa55be6f2ceb9d49: Status 404 returned error can't find the container with id 42e130fca4aa7af6dd9085eb24d9355f16d234aee78e4660fa55be6f2ceb9d49
Nov 02 01:36:09.892560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:09.892537    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"]
Nov 02 01:36:10.160719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:10.160680    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4" event={"ID":"21310443-4ec4-40ae-91da-65292ae655f9","Type":"ContainerStarted","Data":"cdf183c59a2b45fa584c34c8323f8ae62418537980430bb5a35f0f75e5915263"}
Nov 02 01:36:10.160919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:10.160750    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4" event={"ID":"21310443-4ec4-40ae-91da-65292ae655f9","Type":"ContainerStarted","Data":"42e130fca4aa7af6dd9085eb24d9355f16d234aee78e4660fa55be6f2ceb9d49"}
Nov 02 01:36:10.577252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:10.577217    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:11.128505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:11.128467    4851 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:36:11.129170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:11.128540    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:11.169151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:11.169118    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:36:11.211255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:11.211202    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4" podStartSLOduration=3.21118816 podStartE2EDuration="3.21118816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:11.20481633 +0000 UTC m=+1276.509749157" watchObservedRunningTime="2026-11-02 01:36:11.21118816 +0000 UTC m=+1276.516120983"
Nov 02 01:36:11.423330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:11.415517    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:12.286652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:12.286593    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:13.132727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:13.132678    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:13.352372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:13.348874    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:13.938789 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:13.938715    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:14.982513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:14.982478    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:15.199907 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.199879    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:15.217822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.217788    4851 generic.go:356] "Generic (PLEG): container finished" podID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerID="e6e832bda8cd39fac1bb903fff56a2ce4275d404ae25c966249d8b4297db960b" exitCode=0
Nov 02 01:36:15.217992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.217897    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" event={"ID":"0a8d238c-56bd-46bc-b233-c1c0bd398cfe","Type":"ContainerDied","Data":"e6e832bda8cd39fac1bb903fff56a2ce4275d404ae25c966249d8b4297db960b"}
Nov 02 01:36:15.385671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.385645    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:36:15.573938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.573854    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-podinfo\") pod \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") "
Nov 02 01:36:15.574110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.573981    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-socket\") pod \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") "
Nov 02 01:36:15.574110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574076    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istiod-ca-cert\") pod \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") "
Nov 02 01:36:15.574246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574151    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-data\") pod \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") "
Nov 02 01:36:15.574246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574197    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-token\" (UniqueName: \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-token\") pod \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") "
Nov 02 01:36:15.574392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574271    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-certs\") pod \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") "
Nov 02 01:36:15.574392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574347    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-socket" pod "0a8d238c-56bd-46bc-b233-c1c0bd398cfe" (UID: "0a8d238c-56bd-46bc-b233-c1c0bd398cfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:15.574392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574370    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-envoy\") pod \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") "
Nov 02 01:36:15.574553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574476    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-data" pod "0a8d238c-56bd-46bc-b233-c1c0bd398cfe" (UID: "0a8d238c-56bd-46bc-b233-c1c0bd398cfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:15.574553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574492    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-credential-socket\") pod \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") "
Nov 02 01:36:15.574553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574538    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-kube-api-access-wbm87\" (UniqueName: \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-kube-api-access-wbm87\") pod \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\" (UID: \"0a8d238c-56bd-46bc-b233-c1c0bd398cfe\") "
Nov 02 01:36:15.574707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574543    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istiod-ca-cert" pod "0a8d238c-56bd-46bc-b233-c1c0bd398cfe" (UID: "0a8d238c-56bd-46bc-b233-c1c0bd398cfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:36:15.574758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574713    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-certs" pod "0a8d238c-56bd-46bc-b233-c1c0bd398cfe" (UID: "0a8d238c-56bd-46bc-b233-c1c0bd398cfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:15.574863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.574841    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-credential-socket" pod "0a8d238c-56bd-46bc-b233-c1c0bd398cfe" (UID: "0a8d238c-56bd-46bc-b233-c1c0bd398cfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:15.575232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.575217    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:15.575277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.575234    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:15.575277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.575245    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:15.575277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.575254    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:15.575277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.575263    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:15.588213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.588182    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-envoy" pod "0a8d238c-56bd-46bc-b233-c1c0bd398cfe" (UID: "0a8d238c-56bd-46bc-b233-c1c0bd398cfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:15.588391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.588265    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-podinfo" pod "0a8d238c-56bd-46bc-b233-c1c0bd398cfe" (UID: "0a8d238c-56bd-46bc-b233-c1c0bd398cfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:36:15.588464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.588436    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-token" pod "0a8d238c-56bd-46bc-b233-c1c0bd398cfe" (UID: "0a8d238c-56bd-46bc-b233-c1c0bd398cfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:15.588651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.588635    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-kube-api-access-wbm87" pod "0a8d238c-56bd-46bc-b233-c1c0bd398cfe" (UID: "0a8d238c-56bd-46bc-b233-c1c0bd398cfe"). InnerVolumeSpecName "kube-api-access-wbm87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:15.681315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.680938    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:15.681315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.680978    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:15.681315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.680993    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:15.681315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:15.681006    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbm87\" (UniqueName: \"kubernetes.io/projected/0a8d238c-56bd-46bc-b233-c1c0bd398cfe-kube-api-access-wbm87\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:16.228209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.228179    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"
Nov 02 01:36:16.228209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.228189    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh" event={"ID":"0a8d238c-56bd-46bc-b233-c1c0bd398cfe","Type":"ContainerDied","Data":"466d3fab508890f2094c2dbce83f2675c7c5841021672a78ce2d57cea1d156e4"}
Nov 02 01:36:16.228742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.228258    4851 scope.go:122] "RemoveContainer" containerID="e6e832bda8cd39fac1bb903fff56a2ce4275d404ae25c966249d8b4297db960b"
Nov 02 01:36:16.256179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.256152    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"]
Nov 02 01:36:16.261462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.261437    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-tkzdh"]
Nov 02 01:36:16.520112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.520014    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:16.756516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.756483    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6586-b13b99"]
Nov 02 01:36:16.761977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.761952    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy"
Nov 02 01:36:16.761977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.761973    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy"
Nov 02 01:36:16.764703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.764679    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" containerName="istio-proxy"
Nov 02 01:36:16.805232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.804434    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6586-b13b99"]
Nov 02 01:36:16.805232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.804536    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:16.816563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.816533    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vhtnz\""
Nov 02 01:36:16.864728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.864672    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8d238c-56bd-46bc-b233-c1c0bd398cfe" path="/var/lib/kubelet/pods/0a8d238c-56bd-46bc-b233-c1c0bd398cfe/volumes"
Nov 02 01:36:16.926729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:16.926669    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrwn\" (UniqueName: \"kubernetes.io/projected/7e3dded8-6b0b-4d20-8b5f-4e379af059c3-kube-api-access-jkrwn\") pod \"test-curl-6586-b13b99\" (UID: \"7e3dded8-6b0b-4d20-8b5f-4e379af059c3\") " pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:17.030153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:17.030119    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrwn\" (UniqueName: \"kubernetes.io/projected/7e3dded8-6b0b-4d20-8b5f-4e379af059c3-kube-api-access-jkrwn\") pod \"test-curl-6586-b13b99\" (UID: \"7e3dded8-6b0b-4d20-8b5f-4e379af059c3\") " pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:17.046366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:17.046340    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrwn\" (UniqueName: \"kubernetes.io/projected/7e3dded8-6b0b-4d20-8b5f-4e379af059c3-kube-api-access-jkrwn\") pod \"test-curl-6586-b13b99\" (UID: \"7e3dded8-6b0b-4d20-8b5f-4e379af059c3\") " pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:17.200099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:17.200066    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:17.476054 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:17.475975    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:17.531404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:17.531354    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6586-b13b99"]
Nov 02 01:36:17.701192 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:17.701157    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:36:18.336589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:18.336544    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6586-b13b99" event={"ID":"7e3dded8-6b0b-4d20-8b5f-4e379af059c3","Type":"ContainerStarted","Data":"1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423"}
Nov 02 01:36:18.336859 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:18.336841    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6586-b13b99" event={"ID":"7e3dded8-6b0b-4d20-8b5f-4e379af059c3","Type":"ContainerStarted","Data":"62c94919e9cb1112ff5f789687a12c49cbbd055c88da7761e5124a91704fc1ca"}
Nov 02 01:36:18.464020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:18.463872    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:18.984737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:18.984692    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:19.358960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.358844    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:19.575452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.575379    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:19.808138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.808080    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6586-b13b99" podStartSLOduration=3.808065092 podStartE2EDuration="3.808065092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:18.375175691 +0000 UTC m=+1283.680108519" watchObservedRunningTime="2026-11-02 01:36:19.808065092 +0000 UTC m=+1285.112997917"
Nov 02 01:36:19.808387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.808210    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6586-b13b99"]
Nov 02 01:36:19.816207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.816183    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6586-b13b99"]
Nov 02 01:36:19.983485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.983414    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6586-d4eeb2"]
Nov 02 01:36:19.984194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.984171    4851 cpu_manager.go:414] "containerMap: removing container" podUID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" containerName="test-curl-6586-b13b99"
Nov 02 01:36:19.984194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.984189    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" containerName="test-curl-6586-b13b99"
Nov 02 01:36:19.984407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.984317    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" containerName="test-curl-6586-b13b99"
Nov 02 01:36:19.988652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.988548    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:19.993947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:19.993924    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6586-d4eeb2"]
Nov 02 01:36:20.082010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.081979    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:20.088250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.088219    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97ws\" (UniqueName: \"kubernetes.io/projected/fdfb41fb-5a7c-40c8-850c-fdbb6d447560-kube-api-access-z97ws\") pod \"test-curl-6586-d4eeb2\" (UID: \"fdfb41fb-5a7c-40c8-850c-fdbb6d447560\") " pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:20.193821 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.193778    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97ws\" (UniqueName: \"kubernetes.io/projected/fdfb41fb-5a7c-40c8-850c-fdbb6d447560-kube-api-access-z97ws\") pod \"test-curl-6586-d4eeb2\" (UID: \"fdfb41fb-5a7c-40c8-850c-fdbb6d447560\") " pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:20.298873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.298546    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97ws\" (UniqueName: \"kubernetes.io/projected/fdfb41fb-5a7c-40c8-850c-fdbb6d447560-kube-api-access-z97ws\") pod \"test-curl-6586-d4eeb2\" (UID: \"fdfb41fb-5a7c-40c8-850c-fdbb6d447560\") " pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:20.309496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.309462    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:20.358701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.358671    4851 generic.go:356] "Generic (PLEG): container finished" podID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerID="6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194" exitCode=0
Nov 02 01:36:20.358871 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.358746    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" event={"ID":"b0c1d21f-cac3-4641-a6e2-92b2e12551b1","Type":"ContainerDied","Data":"6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194"}
Nov 02 01:36:20.359007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.358988    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6586-b13b99" podUID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" containerName="test-curl-6586-b13b99" containerID="cri-o://1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423" gracePeriod=2
Nov 02 01:36:20.359598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.359363    4851 scope.go:122] "RemoveContainer" containerID="6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194"
Nov 02 01:36:20.361318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.361262    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:20.361429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.361412    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:20.361475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.361453    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:20.390739 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:20.390057    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-b13b99\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:20.401240 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:20.394380    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-b13b99\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:20.559559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.559472    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:20.608223 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:20.608189    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdfb41fb_5a7c_40c8_850c_fdbb6d447560.slice/crio-758af6d8efa8cb4e2b106a65fd5af724b2d5610513b44ffc063e3fab4b238444 WatchSource:0}: Error finding container 758af6d8efa8cb4e2b106a65fd5af724b2d5610513b44ffc063e3fab4b238444: Status 404 returned error can't find the container with id 758af6d8efa8cb4e2b106a65fd5af724b2d5610513b44ffc063e3fab4b238444
Nov 02 01:36:20.611423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.611397    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6586-d4eeb2"]
Nov 02 01:36:20.886242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:20.886158    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:21.373021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:21.372990    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" event={"ID":"b0c1d21f-cac3-4641-a6e2-92b2e12551b1","Type":"ContainerStarted","Data":"36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f"}
Nov 02 01:36:21.374923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:21.374885    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6586-d4eeb2" event={"ID":"fdfb41fb-5a7c-40c8-850c-fdbb6d447560","Type":"ContainerStarted","Data":"7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5"}
Nov 02 01:36:21.374923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:21.374921    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6586-d4eeb2" event={"ID":"fdfb41fb-5a7c-40c8-850c-fdbb6d447560","Type":"ContainerStarted","Data":"758af6d8efa8cb4e2b106a65fd5af724b2d5610513b44ffc063e3fab4b238444"}
Nov 02 01:36:21.419161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:21.419107    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6586-d4eeb2" podStartSLOduration=2.419094263 podStartE2EDuration="2.419094263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:21.41701259 +0000 UTC m=+1286.721945418" watchObservedRunningTime="2026-11-02 01:36:21.419094263 +0000 UTC m=+1286.724027089"
Nov 02 01:36:22.009316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:21.999978    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:36:22.013820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:22.010700    4851 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 01:36:22.013820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:22.010768    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 01:36:22.148508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:22.146187    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:22.201315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:22.192455    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:36:22.513218 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:22.513187    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:23.039848 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.035381    4851 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 01:36:23.039848 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.035436    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 01:36:23.177452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.177274    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6586-d4eeb2"]
Nov 02 01:36:23.179170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.179144    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6586-d4eeb2"]
Nov 02 01:36:23.192721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.186395    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6586-d8e0ba"]
Nov 02 01:36:23.192721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.190460    4851 cpu_manager.go:414] "containerMap: removing container" podUID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" containerName="test-curl-6586-d4eeb2"
Nov 02 01:36:23.192721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.190484    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" containerName="test-curl-6586-d4eeb2"
Nov 02 01:36:23.192721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.191550    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" containerName="test-curl-6586-d4eeb2"
Nov 02 01:36:23.207682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.207651    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6586-d8e0ba"]
Nov 02 01:36:23.208233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.207777    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-d8e0ba"
Nov 02 01:36:23.215788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.215770    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:23.216333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.216270    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:23.337369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.334869    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3dded8-6b0b-4d20-8b5f-4e379af059c3-kube-api-access-jkrwn\" (UniqueName: \"kubernetes.io/projected/7e3dded8-6b0b-4d20-8b5f-4e379af059c3-kube-api-access-jkrwn\") pod \"7e3dded8-6b0b-4d20-8b5f-4e379af059c3\" (UID: \"7e3dded8-6b0b-4d20-8b5f-4e379af059c3\") "
Nov 02 01:36:23.351738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.350493    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nps92\" (UniqueName: \"kubernetes.io/projected/eb6598f2-8b00-402e-b2d5-ac2ba46b76d4-kube-api-access-nps92\") pod \"test-curl-6586-d8e0ba\" (UID: \"eb6598f2-8b00-402e-b2d5-ac2ba46b76d4\") " pod="opendatahub/test-curl-6586-d8e0ba"
Nov 02 01:36:23.382449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.374560    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3dded8-6b0b-4d20-8b5f-4e379af059c3-kube-api-access-jkrwn" pod "7e3dded8-6b0b-4d20-8b5f-4e379af059c3" (UID: "7e3dded8-6b0b-4d20-8b5f-4e379af059c3"). InnerVolumeSpecName "kube-api-access-jkrwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:23.474317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.472546    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nps92\" (UniqueName: \"kubernetes.io/projected/eb6598f2-8b00-402e-b2d5-ac2ba46b76d4-kube-api-access-nps92\") pod \"test-curl-6586-d8e0ba\" (UID: \"eb6598f2-8b00-402e-b2d5-ac2ba46b76d4\") " pod="opendatahub/test-curl-6586-d8e0ba"
Nov 02 01:36:23.474317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.472734    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrwn\" (UniqueName: \"kubernetes.io/projected/7e3dded8-6b0b-4d20-8b5f-4e379af059c3-kube-api-access-jkrwn\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:23.488196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.488159    4851 generic.go:356] "Generic (PLEG): container finished" podID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" containerID="1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423" exitCode=137
Nov 02 01:36:23.488618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.488596    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6586-d4eeb2" podUID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" containerName="test-curl-6586-d4eeb2" containerID="cri-o://7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5" gracePeriod=2
Nov 02 01:36:23.488966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.488948    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:23.489750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.489571    4851 scope.go:122] "RemoveContainer" containerID="1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423"
Nov 02 01:36:23.490180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.489874    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nps92\" (UniqueName: \"kubernetes.io/projected/eb6598f2-8b00-402e-b2d5-ac2ba46b76d4-kube-api-access-nps92\") pod \"test-curl-6586-d8e0ba\" (UID: \"eb6598f2-8b00-402e-b2d5-ac2ba46b76d4\") " pod="opendatahub/test-curl-6586-d8e0ba"
Nov 02 01:36:23.492790 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:23.492755    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-d4eeb2\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:23.495149 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:23.495122    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-d4eeb2\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:23.513900 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:23.513872    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-b13b99\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:23.516242 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:23.516215    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-d4eeb2\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:23.516738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.516725    4851 scope.go:122] "RemoveContainer" containerID="1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423"
Nov 02 01:36:23.517085 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:23.517061    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423\": container with ID starting with 1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423 not found: ID does not exist" containerID="1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423"
Nov 02 01:36:23.517135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.517101    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423"} err="failed to get container status \"1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423\": rpc error: code = NotFound desc = could not find container \"1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423\": container with ID starting with 1e8e3e95048099ab26545baffe3094ee69a551cee3d1eb1ed2f8542dcb8f4423 not found: ID does not exist"
Nov 02 01:36:23.527379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.527356    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-d8e0ba"
Nov 02 01:36:23.978816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.978684    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6586-d8e0ba"]
Nov 02 01:36:23.984851 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:23.984820    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb6598f2_8b00_402e_b2d5_ac2ba46b76d4.slice/crio-f8a481e4cf7326c4b627c1c6207e391dc459b663803371df445f1501d331b0fc WatchSource:0}: Error finding container f8a481e4cf7326c4b627c1c6207e391dc459b663803371df445f1501d331b0fc: Status 404 returned error can't find the container with id f8a481e4cf7326c4b627c1c6207e391dc459b663803371df445f1501d331b0fc
Nov 02 01:36:23.994016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.993392    4851 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 01:36:23.994016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:23.993443    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 01:36:24.076502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:24.076470    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:24.499884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:24.499836    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6586-d8e0ba" event={"ID":"eb6598f2-8b00-402e-b2d5-ac2ba46b76d4","Type":"ContainerStarted","Data":"7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7"}
Nov 02 01:36:24.499884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:24.499878    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6586-d8e0ba" event={"ID":"eb6598f2-8b00-402e-b2d5-ac2ba46b76d4","Type":"ContainerStarted","Data":"f8a481e4cf7326c4b627c1c6207e391dc459b663803371df445f1501d331b0fc"}
Nov 02 01:36:24.519889 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:24.519844    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-b13b99\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:24.520868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:24.520821    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6586-d8e0ba" podStartSLOduration=2.520803887 podStartE2EDuration="2.520803887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:24.517617958 +0000 UTC m=+1289.822550788" watchObservedRunningTime="2026-11-02 01:36:24.520803887 +0000 UTC m=+1289.825736714"
Nov 02 01:36:24.522052 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:24.522022    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-d4eeb2\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:24.866468 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:24.866342    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-b13b99\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" pod="opendatahub/test-curl-6586-b13b99"
Nov 02 01:36:24.867467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:24.867436    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:24.867664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:24.867644    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3dded8-6b0b-4d20-8b5f-4e379af059c3" path="/var/lib/kubelet/pods/7e3dded8-6b0b-4d20-8b5f-4e379af059c3/volumes"
Nov 02 01:36:24.868563 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:24.868544    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-d4eeb2\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:25.033808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.012871    4851 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 01:36:25.033808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.012930    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 01:36:25.099358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.096864    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"]
Nov 02 01:36:25.099358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.098688    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:36:25.103257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.103229    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2c41aa-b5mrt"]
Nov 02 01:36:25.363378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.363344    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6586-d8e0ba"]
Nov 02 01:36:25.389350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.385588    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6586-d8e0ba"]
Nov 02 01:36:25.606160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.606081    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:25.862318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.857377    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:25.993612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.993583    4851 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 01:36:25.993756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:25.993661    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 01:36:26.057827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:26.057525    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:26.074912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:26.074751    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:26.207345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:26.204057    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfb41fb-5a7c-40c8-850c-fdbb6d447560-kube-api-access-z97ws\" (UniqueName: \"kubernetes.io/projected/fdfb41fb-5a7c-40c8-850c-fdbb6d447560-kube-api-access-z97ws\") pod \"fdfb41fb-5a7c-40c8-850c-fdbb6d447560\" (UID: \"fdfb41fb-5a7c-40c8-850c-fdbb6d447560\") "
Nov 02 01:36:26.389548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:26.389485    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfb41fb-5a7c-40c8-850c-fdbb6d447560-kube-api-access-z97ws" pod "fdfb41fb-5a7c-40c8-850c-fdbb6d447560" (UID: "fdfb41fb-5a7c-40c8-850c-fdbb6d447560"). InnerVolumeSpecName "kube-api-access-z97ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:26.476321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:26.461650    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z97ws\" (UniqueName: \"kubernetes.io/projected/fdfb41fb-5a7c-40c8-850c-fdbb6d447560-kube-api-access-z97ws\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:27.198718 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.198672    4851 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 01:36:27.199392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.199345    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 01:36:27.380808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.338437    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:27.381535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.381496    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:27.508183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.508065    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:27.517157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.517110    4851 generic.go:356] "Generic (PLEG): container finished" podID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" containerID="7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5" exitCode=137
Nov 02 01:36:27.517447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.517277    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6586-d8e0ba" podUID="eb6598f2-8b00-402e-b2d5-ac2ba46b76d4" containerName="test-curl-6586-d8e0ba" containerID="cri-o://7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7" gracePeriod=2
Nov 02 01:36:27.517718 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.517698    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-d4eeb2"
Nov 02 01:36:27.526613 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:27.526555    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-d8e0ba\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="eb6598f2-8b00-402e-b2d5-ac2ba46b76d4" pod="opendatahub/test-curl-6586-d8e0ba"
Nov 02 01:36:27.530413 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:27.530355    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-d8e0ba\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="eb6598f2-8b00-402e-b2d5-ac2ba46b76d4" pod="opendatahub/test-curl-6586-d8e0ba"
Nov 02 01:36:27.570396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.570278    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="471fa604-d12d-44d4-878f-8d6138a47835" path="/var/lib/kubelet/pods/471fa604-d12d-44d4-878f-8d6138a47835/volumes"
Nov 02 01:36:27.571469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.570980    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfb41fb-5a7c-40c8-850c-fdbb6d447560" path="/var/lib/kubelet/pods/fdfb41fb-5a7c-40c8-850c-fdbb6d447560/volumes"
Nov 02 01:36:27.573638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.573582    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"]
Nov 02 01:36:27.573888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.573870    4851 scope.go:122] "RemoveContainer" containerID="7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5"
Nov 02 01:36:27.574771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.574712    4851 cpu_manager.go:414] "containerMap: removing container" podUID="eb6598f2-8b00-402e-b2d5-ac2ba46b76d4" containerName="test-curl-6586-d8e0ba"
Nov 02 01:36:27.574771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.574772    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb6598f2-8b00-402e-b2d5-ac2ba46b76d4" containerName="test-curl-6586-d8e0ba"
Nov 02 01:36:27.575079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.575064    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb6598f2-8b00-402e-b2d5-ac2ba46b76d4" containerName="test-curl-6586-d8e0ba"
Nov 02 01:36:27.581093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.581067    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"]
Nov 02 01:36:27.581253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.581169    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.588845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.588802    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-41215f-openshift-default-dockercfg-btcxv\""
Nov 02 01:36:27.593807 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.593706    4851 scope.go:122] "RemoveContainer" containerID="7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5"
Nov 02 01:36:27.594116 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:27.594086    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5\": container with ID starting with 7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5 not found: ID does not exist" containerID="7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5"
Nov 02 01:36:27.594194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.594128    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5"} err="failed to get container status \"7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5\": rpc error: code = NotFound desc = could not find container \"7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5\": container with ID starting with 7751b9abd915f5ec25fa80a8696885d05b897b5f84317e64dc8e2f57c4f71af5 not found: ID does not exist"
Nov 02 01:36:27.599578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.599544    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:27.625703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.625671    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-certs\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.625703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.625709    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-data\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.625943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.625888    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kw2s\" (UniqueName: \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-kube-api-access-4kw2s\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.626138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.626111    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-token\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.626197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.626160    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-credential-socket\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.626197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.626189    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-envoy\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.626387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.626366    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-podinfo\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.626458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.626444    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-socket\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.626528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.626509    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istiod-ca-cert\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.747646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.747603    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-certs\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.791234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.791138    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-data\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.793476    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kw2s\" (UniqueName: \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-kube-api-access-4kw2s\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.793730    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-token\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.793790    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-credential-socket\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.793822    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-envoy\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.748310    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-certs\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.793907    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-podinfo\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.793993    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-socket\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.794032    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istiod-ca-cert\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.794774    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-credential-socket\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.794908    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istiod-ca-cert\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.795150    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-socket\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.796861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.792648    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-data\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.799120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.799086    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-envoy\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.799498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.799476    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-podinfo\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.807243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.807212    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-token\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.807985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.807960    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kw2s\" (UniqueName: \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-kube-api-access-4kw2s\") pod \"e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") " pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:27.868579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.868546    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:27.910029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:27.909995    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:28.227761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.227732    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:36:28.230096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.230069    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:36:28.364280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.364252    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"]
Nov 02 01:36:28.364833 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:28.364805    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bda2b76_d865_47d7_9c84_e01ffbd776f4.slice/crio-47f62ca65caf0094573d5bb734fc8432badc406ebbaef26f36fd216a8d8001a5 WatchSource:0}: Error finding container 47f62ca65caf0094573d5bb734fc8432badc406ebbaef26f36fd216a8d8001a5: Status 404 returned error can't find the container with id 47f62ca65caf0094573d5bb734fc8432badc406ebbaef26f36fd216a8d8001a5
Nov 02 01:36:28.368254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.368216    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:28.368371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.368325    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:28.368411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.368376    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:28.530043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.529999    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" event={"ID":"6bda2b76-d865-47d7-9c84-e01ffbd776f4","Type":"ContainerStarted","Data":"eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80"}
Nov 02 01:36:28.530237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.530077    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" event={"ID":"6bda2b76-d865-47d7-9c84-e01ffbd776f4","Type":"ContainerStarted","Data":"47f62ca65caf0094573d5bb734fc8432badc406ebbaef26f36fd216a8d8001a5"}
Nov 02 01:36:28.561696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.561646    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" podStartSLOduration=2.561630827 podStartE2EDuration="2.561630827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:28.556394491 +0000 UTC m=+1293.861327328" watchObservedRunningTime="2026-11-02 01:36:28.561630827 +0000 UTC m=+1293.866563712"
Nov 02 01:36:28.686266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.686238    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:28.918316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:28.910311    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:29.409388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:29.409352    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:29.809455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:29.809428    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-d8e0ba"
Nov 02 01:36:29.917010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:29.916979    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:29.945505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:29.945477    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb6598f2-8b00-402e-b2d5-ac2ba46b76d4-kube-api-access-nps92\" (UniqueName: \"kubernetes.io/projected/eb6598f2-8b00-402e-b2d5-ac2ba46b76d4-kube-api-access-nps92\") pod \"eb6598f2-8b00-402e-b2d5-ac2ba46b76d4\" (UID: \"eb6598f2-8b00-402e-b2d5-ac2ba46b76d4\") "
Nov 02 01:36:29.947784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:29.947754    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb6598f2-8b00-402e-b2d5-ac2ba46b76d4-kube-api-access-nps92" pod "eb6598f2-8b00-402e-b2d5-ac2ba46b76d4" (UID: "eb6598f2-8b00-402e-b2d5-ac2ba46b76d4"). InnerVolumeSpecName "kube-api-access-nps92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:30.048565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.048533    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nps92\" (UniqueName: \"kubernetes.io/projected/eb6598f2-8b00-402e-b2d5-ac2ba46b76d4-kube-api-access-nps92\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:30.440568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.440539    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:30.634369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.634280    4851 generic.go:356] "Generic (PLEG): container finished" podID="eb6598f2-8b00-402e-b2d5-ac2ba46b76d4" containerID="7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7" exitCode=137
Nov 02 01:36:30.634369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.634374    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6586-d8e0ba"
Nov 02 01:36:30.634609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.634411    4851 scope.go:122] "RemoveContainer" containerID="7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7"
Nov 02 01:36:30.634846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.634827    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:30.636495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.636452    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:36:30.646856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.646841    4851 scope.go:122] "RemoveContainer" containerID="7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7"
Nov 02 01:36:30.647145 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:30.647126    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7\": container with ID starting with 7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7 not found: ID does not exist" containerID="7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7"
Nov 02 01:36:30.647191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.647152    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7"} err="failed to get container status \"7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7\": rpc error: code = NotFound desc = could not find container \"7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7\": container with ID starting with 7dcf60aba86800a08a243aec5c8bf16eaba40dcb3661554b85c984c944a17cb7 not found: ID does not exist"
Nov 02 01:36:30.665473 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:30.665443    4851 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6586-d8e0ba\" is forbidden: User \"system:node:ip-10-0-140-23.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-23.us-east-2.compute.internal' and this object" podUID="eb6598f2-8b00-402e-b2d5-ac2ba46b76d4" pod="opendatahub/test-curl-6586-d8e0ba"
Nov 02 01:36:30.856777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.856703    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb6598f2-8b00-402e-b2d5-ac2ba46b76d4" path="/var/lib/kubelet/pods/eb6598f2-8b00-402e-b2d5-ac2ba46b76d4/volumes"
Nov 02 01:36:30.863632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:30.863603    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:31.007991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.007956    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"]
Nov 02 01:36:31.014178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.014158    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.018700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.018674    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:36:31.018864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.018722    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:36:31.018864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.018830    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:36:31.018986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.018899    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:36:31.022021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.022000    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"]
Nov 02 01:36:31.067090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.067056    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.067272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.067244    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.067451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.067429    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4dkw\" (UniqueName: \"kubernetes.io/projected/23d4e564-ddd9-4415-9823-6e39b5c314f3-kube-api-access-z4dkw\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.098221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.098195    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:31.169156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.169127    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.169366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.169253    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.169461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.169368    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dkw\" (UniqueName: \"kubernetes.io/projected/23d4e564-ddd9-4415-9823-6e39b5c314f3-kube-api-access-z4dkw\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.169983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.169958    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.171875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.171853    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.182064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.182030    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dkw\" (UniqueName: \"kubernetes.io/projected/23d4e564-ddd9-4415-9823-6e39b5c314f3-kube-api-access-z4dkw\") pod \"maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.334543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.334509    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:31.541179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.541100    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:31.744596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:31.744554    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"]
Nov 02 01:36:32.044400 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.042472    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:32.090598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.090566    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"]
Nov 02 01:36:32.097089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.097062    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:36:32.097383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.097348    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"]
Nov 02 01:36:32.102159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.102129    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:36:32.107753 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.107728    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"]
Nov 02 01:36:32.135385 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.135353    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"]
Nov 02 01:36:32.159783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.159758    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-41215f\""
Nov 02 01:36:32.200355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.200305    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578eb675-6e20-4ed1-a546-a0bb01f3f899-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds\" (UID: \"578eb675-6e20-4ed1-a546-a0bb01f3f899\") " pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:36:32.200525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.200423    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsfrf\" (UniqueName: \"kubernetes.io/projected/2ec4c675-7706-4edf-808a-a542b0986037-kube-api-access-zsfrf\") pod \"payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5\" (UID: \"2ec4c675-7706-4edf-808a-a542b0986037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:36:32.200599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.200584    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4twh\" (UniqueName: \"kubernetes.io/projected/578eb675-6e20-4ed1-a546-a0bb01f3f899-kube-api-access-d4twh\") pod \"payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds\" (UID: \"578eb675-6e20-4ed1-a546-a0bb01f3f899\") " pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:36:32.200711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.200697    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec4c675-7706-4edf-808a-a542b0986037-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5\" (UID: \"2ec4c675-7706-4edf-808a-a542b0986037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:36:32.286444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.283450    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:32.309006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.308909    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4twh\" (UniqueName: \"kubernetes.io/projected/578eb675-6e20-4ed1-a546-a0bb01f3f899-kube-api-access-d4twh\") pod \"payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds\" (UID: \"578eb675-6e20-4ed1-a546-a0bb01f3f899\") " pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:36:32.314846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.314815    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec4c675-7706-4edf-808a-a542b0986037-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5\" (UID: \"2ec4c675-7706-4edf-808a-a542b0986037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:36:32.340316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.326825    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578eb675-6e20-4ed1-a546-a0bb01f3f899-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds\" (UID: \"578eb675-6e20-4ed1-a546-a0bb01f3f899\") " pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:36:32.342459 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.342432    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsfrf\" (UniqueName: \"kubernetes.io/projected/2ec4c675-7706-4edf-808a-a542b0986037-kube-api-access-zsfrf\") pod \"payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5\" (UID: \"2ec4c675-7706-4edf-808a-a542b0986037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:36:32.343122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.341498    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578eb675-6e20-4ed1-a546-a0bb01f3f899-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds\" (UID: \"578eb675-6e20-4ed1-a546-a0bb01f3f899\") " pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:36:32.343311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.326572    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec4c675-7706-4edf-808a-a542b0986037-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5\" (UID: \"2ec4c675-7706-4edf-808a-a542b0986037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:36:32.367799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.358059    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4twh\" (UniqueName: \"kubernetes.io/projected/578eb675-6e20-4ed1-a546-a0bb01f3f899-kube-api-access-d4twh\") pod \"payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds\" (UID: \"578eb675-6e20-4ed1-a546-a0bb01f3f899\") " pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:36:32.390559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.384155    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsfrf\" (UniqueName: \"kubernetes.io/projected/2ec4c675-7706-4edf-808a-a542b0986037-kube-api-access-zsfrf\") pod \"payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5\" (UID: \"2ec4c675-7706-4edf-808a-a542b0986037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:36:32.426381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.420607    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:36:32.442068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.441832    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:36:32.758632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.758593    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj" event={"ID":"23d4e564-ddd9-4415-9823-6e39b5c314f3","Type":"ContainerStarted","Data":"27e698823a47da5ebc17e07a1917d06213c78105abad3babd774dd02a1226905"}
Nov 02 01:36:32.759018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.758650    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj" event={"ID":"23d4e564-ddd9-4415-9823-6e39b5c314f3","Type":"ContainerStarted","Data":"59b04ef93e336fac04afe3ba110f5b64f5d32ec217f7be307723102010acbadf"}
Nov 02 01:36:32.783884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.783807    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj" podStartSLOduration=2.783784159 podStartE2EDuration="2.783784159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:32.780701749 +0000 UTC m=+1298.085634576" watchObservedRunningTime="2026-11-02 01:36:32.783784159 +0000 UTC m=+1298.088716986"
Nov 02 01:36:32.961756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:32.961699    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:33.146996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.146959    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"]
Nov 02 01:36:33.151864 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:33.151837    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec4c675_7706_4edf_808a_a542b0986037.slice/crio-b68a5e85fa7c34d5a1820a28cb89a7175db709f13029be4861d45f5cee3b903f WatchSource:0}: Error finding container b68a5e85fa7c34d5a1820a28cb89a7175db709f13029be4861d45f5cee3b903f: Status 404 returned error can't find the container with id b68a5e85fa7c34d5a1820a28cb89a7175db709f13029be4861d45f5cee3b903f
Nov 02 01:36:33.159608 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:33.159567    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578eb675_6e20_4ed1_a546_a0bb01f3f899.slice/crio-06b290d0c009374f967ad4a283dab02ed9dfbe4d55bd1ebbd26a06df105e6c36 WatchSource:0}: Error finding container 06b290d0c009374f967ad4a283dab02ed9dfbe4d55bd1ebbd26a06df105e6c36: Status 404 returned error can't find the container with id 06b290d0c009374f967ad4a283dab02ed9dfbe4d55bd1ebbd26a06df105e6c36
Nov 02 01:36:33.161826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.161804    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"]
Nov 02 01:36:33.767994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.767953    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds" event={"ID":"578eb675-6e20-4ed1-a546-a0bb01f3f899","Type":"ContainerStarted","Data":"44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f"}
Nov 02 01:36:33.768514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.768046    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:36:33.768514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.768070    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds" event={"ID":"578eb675-6e20-4ed1-a546-a0bb01f3f899","Type":"ContainerStarted","Data":"06b290d0c009374f967ad4a283dab02ed9dfbe4d55bd1ebbd26a06df105e6c36"}
Nov 02 01:36:33.769784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.769758    4851 generic.go:356] "Generic (PLEG): container finished" podID="23d4e564-ddd9-4415-9823-6e39b5c314f3" containerID="27e698823a47da5ebc17e07a1917d06213c78105abad3babd774dd02a1226905" exitCode=0
Nov 02 01:36:33.769919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.769830    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj" event={"ID":"23d4e564-ddd9-4415-9823-6e39b5c314f3","Type":"ContainerDied","Data":"27e698823a47da5ebc17e07a1917d06213c78105abad3babd774dd02a1226905"}
Nov 02 01:36:33.771560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.771530    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5" event={"ID":"2ec4c675-7706-4edf-808a-a542b0986037","Type":"ContainerStarted","Data":"b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001"}
Nov 02 01:36:33.771670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.771567    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5" event={"ID":"2ec4c675-7706-4edf-808a-a542b0986037","Type":"ContainerStarted","Data":"b68a5e85fa7c34d5a1820a28cb89a7175db709f13029be4861d45f5cee3b903f"}
Nov 02 01:36:33.771670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.771609    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:36:33.791377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.791317    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds" podStartSLOduration=1.791304515 podStartE2EDuration="1.791304515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:33.786231988 +0000 UTC m=+1299.091164814" watchObservedRunningTime="2026-11-02 01:36:33.791304515 +0000 UTC m=+1299.096237335"
Nov 02 01:36:33.801231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.801205    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:33.830980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:33.830942    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5" podStartSLOduration=1.830931334 podStartE2EDuration="1.830931334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:33.805371639 +0000 UTC m=+1299.110304467" watchObservedRunningTime="2026-11-02 01:36:33.830931334 +0000 UTC m=+1299.135864160"
Nov 02 01:36:34.571190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:34.571158    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:34.969575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:34.969548    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:34.996390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:34.996310    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-xks-ca\") pod \"23d4e564-ddd9-4415-9823-6e39b5c314f3\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") "
Nov 02 01:36:34.996390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:34.996382    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d4e564-ddd9-4415-9823-6e39b5c314f3-kube-api-access-z4dkw\" (UniqueName: \"kubernetes.io/projected/23d4e564-ddd9-4415-9823-6e39b5c314f3-kube-api-access-z4dkw\") pod \"23d4e564-ddd9-4415-9823-6e39b5c314f3\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") "
Nov 02 01:36:34.996644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:34.996507    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-openshift-ca\") pod \"23d4e564-ddd9-4415-9823-6e39b5c314f3\" (UID: \"23d4e564-ddd9-4415-9823-6e39b5c314f3\") "
Nov 02 01:36:34.996867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:34.996843    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-openshift-ca" pod "23d4e564-ddd9-4415-9823-6e39b5c314f3" (UID: "23d4e564-ddd9-4415-9823-6e39b5c314f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:36:34.997151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:34.997131    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:34.998724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:34.998695    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-xks-ca" pod "23d4e564-ddd9-4415-9823-6e39b5c314f3" (UID: "23d4e564-ddd9-4415-9823-6e39b5c314f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:36:34.999631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:34.999610    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d4e564-ddd9-4415-9823-6e39b5c314f3-kube-api-access-z4dkw" pod "23d4e564-ddd9-4415-9823-6e39b5c314f3" (UID: "23d4e564-ddd9-4415-9823-6e39b5c314f3"). InnerVolumeSpecName "kube-api-access-z4dkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:35.099228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:35.099146    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d4e564-ddd9-4415-9823-6e39b5c314f3-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:35.099228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:35.099180    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4dkw\" (UniqueName: \"kubernetes.io/projected/23d4e564-ddd9-4415-9823-6e39b5c314f3-kube-api-access-z4dkw\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:35.557325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:35.555155    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:35.820926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:35.820832    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj" event={"ID":"23d4e564-ddd9-4415-9823-6e39b5c314f3","Type":"ContainerDied","Data":"59b04ef93e336fac04afe3ba110f5b64f5d32ec217f7be307723102010acbadf"}
Nov 02 01:36:35.820926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:35.820893    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59b04ef93e336fac04afe3ba110f5b64f5d32ec217f7be307723102010acbadf"
Nov 02 01:36:35.821169 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:35.820961    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"
Nov 02 01:36:36.134158 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:36.134078    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:36.604391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:36.604355    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:36.829095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:36.829051    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:37.080742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:37.080706    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:37.333364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:37.330837    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:38.135865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:38.135829    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:38.713235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:38.713194    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:39.213268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:39.213220    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:39.816191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:39.816159    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"]
Nov 02 01:36:39.816758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:39.816435    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy" containerID="cri-o://cd0b18a5667aad25db9afd9b677900e4430d2363921dde4c082e42b5ff77d468" gracePeriod=30
Nov 02 01:36:39.869768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:39.869735    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"]
Nov 02 01:36:39.869992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:39.869971    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" containerID="cri-o://530fa39b156027983bdd36ab12fa28256897addba275241680b867749d73f78c" gracePeriod=30
Nov 02 01:36:39.879150 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:39.879121    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"]
Nov 02 01:36:39.879386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:39.879363    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" containerID="cri-o://3680c7f5c4e5145d060355918aac14b95c1c0ee3bc619e08f96049285c927404" gracePeriod=30
Nov 02 01:36:40.012180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:40.012143    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"]
Nov 02 01:36:40.012420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:40.012385    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy" containerID="cri-o://763f1dc8bf4d638ff7e266004c26036551a8cda3bf4a54bbcf547b0a0dca0ffa" gracePeriod=30
Nov 02 01:36:40.176963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:40.176927    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"]
Nov 02 01:36:40.177211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:40.177159    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy" containerID="cri-o://ad6a3825f1f3291edcdd94abfd5bf160d89757355ae8b500e374e08c80c37d69" gracePeriod=30
Nov 02 01:36:40.261598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:40.261571    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"]
Nov 02 01:36:40.261797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:40.261767    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy" containerID="cri-o://1ca66c6a16d173d4844902705250627ccf6eb79c453d43f0bf651bd750ef7ab0" gracePeriod=30
Nov 02 01:36:40.266545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:40.266521    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"]
Nov 02 01:36:40.267086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:40.267060    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy" containerID="cri-o://9c27755317ff9ab84e50a4404e328bcb0704761a71fde910c9975e7f7c81e582" gracePeriod=30
Nov 02 01:36:40.299664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:40.299628    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:40.643271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:40.643196    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:41.140024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:41.139991    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"]
Nov 02 01:36:41.196780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:41.196697    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:41.439329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:41.439274    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:42.422312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:42.416610    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:36:42.422312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:42.416682    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:36:42.465314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:42.463372    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:36:42.465314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:42.463442    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:36:42.540077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:42.539985    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:44.217087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.217044    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:44.402312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.400759    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:36:44.402312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.401150    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:36:44.418186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.417333    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:44.516471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.516374    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:36:44.566317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.487307    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:44.609691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.608382    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:36:44.664167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.662784    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:36:44.679415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.679365    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:44.788156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.787049    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:44.822316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.797534    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:36:44.852429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.851480    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:36:44.939923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.915385    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:44.939923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.915647    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:36:44.999327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:44.991838    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:45.090349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.081372    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:45.416317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.407357    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:45.478151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.452538    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:36:45.625333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.625270    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:45.671779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.670561    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:45.679855 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.679808    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:45.766739 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.751771    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:36:45.766739 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.752069    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:36:45.819206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.819156    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:45.819871 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.819807    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:45.961633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.961587    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:45.972617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:45.972555    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:46.108987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.104551    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:46.130082    4851 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.245s"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130175    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130318    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130602    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130660    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130695    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130731    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130775    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130809    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130841    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130871    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130903    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130930    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130959    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.130985    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.131014    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.131044    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.131077    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.131109    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:36:46.157552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.131135    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:36:46.169337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.160696    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:36:46.189283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.187391    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk" podUID="10c4b872-5d76-4900-b4ea-09331641a617" containerName="payload-pre-processing" containerID="cri-o://3a5fdf2bd6a64649919a647a1f3d18c7b9bd7ae7e508581d21a8e39de92ee9a7" gracePeriod=25
Nov 02 01:36:46.517946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.510722    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:46.591496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.584368    4851 generic.go:356] "Generic (PLEG): container finished" podID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerID="cd0b18a5667aad25db9afd9b677900e4430d2363921dde4c082e42b5ff77d468" exitCode=0
Nov 02 01:36:46.591496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.584596    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" event={"ID":"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b","Type":"ContainerDied","Data":"cd0b18a5667aad25db9afd9b677900e4430d2363921dde4c082e42b5ff77d468"}
Nov 02 01:36:46.718408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.718157    4851 generic.go:356] "Generic (PLEG): container finished" podID="dd465d06-c445-4b09-a15f-5d97797c850c" containerID="1ca66c6a16d173d4844902705250627ccf6eb79c453d43f0bf651bd750ef7ab0" exitCode=0
Nov 02 01:36:46.718408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.718281    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" event={"ID":"dd465d06-c445-4b09-a15f-5d97797c850c","Type":"ContainerDied","Data":"1ca66c6a16d173d4844902705250627ccf6eb79c453d43f0bf651bd750ef7ab0"}
Nov 02 01:36:46.775714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.775488    4851 generic.go:356] "Generic (PLEG): container finished" podID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerID="763f1dc8bf4d638ff7e266004c26036551a8cda3bf4a54bbcf547b0a0dca0ffa" exitCode=0
Nov 02 01:36:46.775714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.775586    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" event={"ID":"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524","Type":"ContainerDied","Data":"763f1dc8bf4d638ff7e266004c26036551a8cda3bf4a54bbcf547b0a0dca0ffa"}
Nov 02 01:36:46.780151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.779598    4851 generic.go:356] "Generic (PLEG): container finished" podID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerID="530fa39b156027983bdd36ab12fa28256897addba275241680b867749d73f78c" exitCode=0
Nov 02 01:36:46.780151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.779700    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" event={"ID":"e8c4b040-6e11-4236-b7f2-9438685e5ba3","Type":"ContainerDied","Data":"530fa39b156027983bdd36ab12fa28256897addba275241680b867749d73f78c"}
Nov 02 01:36:46.810268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.810234    4851 generic.go:356] "Generic (PLEG): container finished" podID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerID="3680c7f5c4e5145d060355918aac14b95c1c0ee3bc619e08f96049285c927404" exitCode=0
Nov 02 01:36:46.820573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.813082    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" event={"ID":"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e","Type":"ContainerDied","Data":"3680c7f5c4e5145d060355918aac14b95c1c0ee3bc619e08f96049285c927404"}
Nov 02 01:36:46.974046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:46.942511    4851 generic.go:356] "Generic (PLEG): container finished" podID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerID="9c27755317ff9ab84e50a4404e328bcb0704761a71fde910c9975e7f7c81e582" exitCode=0
Nov 02 01:36:47.031773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:47.031400    4851 generic.go:356] "Generic (PLEG): container finished" podID="3d43d33e-3b63-4c69-ae89-480cef886400" containerID="ad6a3825f1f3291edcdd94abfd5bf160d89757355ae8b500e374e08c80c37d69" exitCode=0
Nov 02 01:36:47.194376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:47.179969    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:47.341322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:47.332908    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" event={"ID":"cf435d77-26a8-436f-b3fc-b0a1235bd325","Type":"ContainerDied","Data":"9c27755317ff9ab84e50a4404e328bcb0704761a71fde910c9975e7f7c81e582"}
Nov 02 01:36:47.343219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:47.343118    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" event={"ID":"3d43d33e-3b63-4c69-ae89-480cef886400","Type":"ContainerDied","Data":"ad6a3825f1f3291edcdd94abfd5bf160d89757355ae8b500e374e08c80c37d69"}
Nov 02 01:36:47.723682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:47.723603    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:48.081754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.081346    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:36:48.082443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.082411    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd" event={"ID":"3d43d33e-3b63-4c69-ae89-480cef886400","Type":"ContainerDied","Data":"411d5f9853168075b3c46e20be07510a81a228490f4e3462c5548e572498774a"}
Nov 02 01:36:48.082660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.082642    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="411d5f9853168075b3c46e20be07510a81a228490f4e3462c5548e572498774a"
Nov 02 01:36:48.088570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.088544    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92" event={"ID":"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b","Type":"ContainerDied","Data":"0e21ab406f96b196565e68799226cf45ce2390745f3be89669b3e5d23bbe5699"}
Nov 02 01:36:48.088773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.088756    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e21ab406f96b196565e68799226cf45ce2390745f3be89669b3e5d23bbe5699"
Nov 02 01:36:48.091199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.091176    4851 generic.go:356] "Generic (PLEG): container finished" podID="10c4b872-5d76-4900-b4ea-09331641a617" containerID="3a5fdf2bd6a64649919a647a1f3d18c7b9bd7ae7e508581d21a8e39de92ee9a7" exitCode=0
Nov 02 01:36:48.091507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.091486    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk" event={"ID":"10c4b872-5d76-4900-b4ea-09331641a617","Type":"ContainerDied","Data":"3a5fdf2bd6a64649919a647a1f3d18c7b9bd7ae7e508581d21a8e39de92ee9a7"}
Nov 02 01:36:48.093500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.093474    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6" event={"ID":"dd465d06-c445-4b09-a15f-5d97797c850c","Type":"ContainerDied","Data":"71750d215cd0fff425a8a6cd63f9edaf81d0a57e023d817998a60eb0970008e4"}
Nov 02 01:36:48.093683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.093668    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71750d215cd0fff425a8a6cd63f9edaf81d0a57e023d817998a60eb0970008e4"
Nov 02 01:36:48.095341 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:36:48.095318    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:36:48.096934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.096913    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" event={"ID":"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524","Type":"ContainerDied","Data":"aa53bd191f8c881c636cd0c836566f58a77bae0f03668510d6db2f4f59b6c13f"}
Nov 02 01:36:48.097088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.097073    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa53bd191f8c881c636cd0c836566f58a77bae0f03668510d6db2f4f59b6c13f"
Nov 02 01:36:48.100528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.100498    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25" event={"ID":"e8c4b040-6e11-4236-b7f2-9438685e5ba3","Type":"ContainerDied","Data":"43ae94b8731a8cb47a9f65381778af2fcc4308d4ad92c951a2eac0328f3d6465"}
Nov 02 01:36:48.101439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.101420    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43ae94b8731a8cb47a9f65381778af2fcc4308d4ad92c951a2eac0328f3d6465"
Nov 02 01:36:48.102492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.100969    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:48.146678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.146638    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6" event={"ID":"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e","Type":"ContainerDied","Data":"87b57d8f646216aa17a616530878f6b5cdcfcde21b648730b3e1b484214f1a53"}
Nov 02 01:36:48.147253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.147226    4851 scope.go:122] "RemoveContainer" containerID="3680c7f5c4e5145d060355918aac14b95c1c0ee3bc619e08f96049285c927404"
Nov 02 01:36:48.190432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.190403    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:36:48.191806 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.191787    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256058    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-certs\") pod \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256152    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-token\" (UniqueName: \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-token\") pod \"cf435d77-26a8-436f-b3fc-b0a1235bd325\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256262    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf435d77-26a8-436f-b3fc-b0a1235bd325-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf435d77-26a8-436f-b3fc-b0a1235bd325-istiod-ca-cert\") pod \"cf435d77-26a8-436f-b3fc-b0a1235bd325\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256313    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-credential-socket\") pod \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256384    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-credential-socket\") pod \"cf435d77-26a8-436f-b3fc-b0a1235bd325\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256480    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-data\") pod \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256547    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-podinfo\") pod \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256621    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-kube-api-access-ws8lk\" (UniqueName: \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-kube-api-access-ws8lk\") pod \"cf435d77-26a8-436f-b3fc-b0a1235bd325\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256704    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-podinfo\") pod \"cf435d77-26a8-436f-b3fc-b0a1235bd325\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256751    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-certs\") pod \"cf435d77-26a8-436f-b3fc-b0a1235bd325\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256807    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-envoy\") pod \"cf435d77-26a8-436f-b3fc-b0a1235bd325\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256870    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istiod-ca-cert\") pod \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256908    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-envoy\") pod \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.256961    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-kube-api-access-n4tjl\" (UniqueName: \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-kube-api-access-n4tjl\") pod \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.257085    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-socket\") pod \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.257227    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-data\") pod \"cf435d77-26a8-436f-b3fc-b0a1235bd325\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.257337    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-token\" (UniqueName: \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-token\") pod \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") "
Nov 02 01:36:48.263373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.257420    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-socket\") pod \"cf435d77-26a8-436f-b3fc-b0a1235bd325\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") "
Nov 02 01:36:48.264428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.264191    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd" event={"ID":"cf435d77-26a8-436f-b3fc-b0a1235bd325","Type":"ContainerDied","Data":"cf3ada1ee33b7d75b5a16a405493f70bc1bfee4206a1a3aa55734a2223261f71"}
Nov 02 01:36:48.276428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.267233    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-socket" pod "cf435d77-26a8-436f-b3fc-b0a1235bd325" (UID: "cf435d77-26a8-436f-b3fc-b0a1235bd325"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.276428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.267536    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-certs" pod "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" (UID: "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.276428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.268154    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:36:48.276428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.268688    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-certs" pod "cf435d77-26a8-436f-b3fc-b0a1235bd325" (UID: "cf435d77-26a8-436f-b3fc-b0a1235bd325"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.276428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.269006    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf435d77-26a8-436f-b3fc-b0a1235bd325-istiod-ca-cert" pod "cf435d77-26a8-436f-b3fc-b0a1235bd325" (UID: "cf435d77-26a8-436f-b3fc-b0a1235bd325"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:36:48.276428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.269347    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-credential-socket" pod "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" (UID: "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.276428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.269580    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-credential-socket" pod "cf435d77-26a8-436f-b3fc-b0a1235bd325" (UID: "cf435d77-26a8-436f-b3fc-b0a1235bd325"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.276428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.269800    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-data" pod "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" (UID: "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.303015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.302980    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istiod-ca-cert" pod "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" (UID: "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:36:48.414113 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.413769    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-data" pod "cf435d77-26a8-436f-b3fc-b0a1235bd325" (UID: "cf435d77-26a8-436f-b3fc-b0a1235bd325"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.421400 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.419434    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:36:48.498208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.496961    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-envoy" pod "cf435d77-26a8-436f-b3fc-b0a1235bd325" (UID: "cf435d77-26a8-436f-b3fc-b0a1235bd325"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.499387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.499260    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.499387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.499313    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.499387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.499329    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf435d77-26a8-436f-b3fc-b0a1235bd325-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.499387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.499346    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.500349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.499699    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.500349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.499726    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.500349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.499740    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.500349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.499756    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.500349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.499784    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.500349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.499803    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.501497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.500722    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:36:48.502165 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.502144    4851 scope.go:122] "RemoveContainer" containerID="9c27755317ff9ab84e50a4404e328bcb0704761a71fde910c9975e7f7c81e582"
Nov 02 01:36:48.513746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.513659    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-token" pod "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" (UID: "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.515217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.513886    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-socket" pod "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" (UID: "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.612186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.610780    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-token\") pod \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") "
Nov 02 01:36:48.612186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.610847    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-envoy\") pod \"3d43d33e-3b63-4c69-ae89-480cef886400\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") "
Nov 02 01:36:48.612186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.610906    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-kube-api-access-vh7p4\" (UniqueName: \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-kube-api-access-vh7p4\") pod \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") "
Nov 02 01:36:48.612186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.610941    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-kube-api-access-8kj4d\" (UniqueName: \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-kube-api-access-8kj4d\") pod \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") "
Nov 02 01:36:48.612186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.611029    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-podinfo\") pod \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") "
Nov 02 01:36:48.612186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.611101    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-credential-socket\") pod \"3d43d33e-3b63-4c69-ae89-480cef886400\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") "
Nov 02 01:36:48.612186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.611179    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-credential-socket\") pod \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") "
Nov 02 01:36:48.612186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.611219    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-certs\") pod \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") "
Nov 02 01:36:48.614629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.613144    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-envoy" pod "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" (UID: "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.614629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.613260    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-podinfo" pod "cf435d77-26a8-436f-b3fc-b0a1235bd325" (UID: "cf435d77-26a8-436f-b3fc-b0a1235bd325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615051    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-envoy\") pod \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615122    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-certs\") pod \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615199    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-podinfo\") pod \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615245    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-data\") pod \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615328    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-data\") pod \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615421    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d43d33e-3b63-4c69-ae89-480cef886400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d43d33e-3b63-4c69-ae89-480cef886400-istio-podinfo\") pod \"3d43d33e-3b63-4c69-ae89-480cef886400\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615463    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istiod-ca-cert\") pod \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615502    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-envoy\") pod \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615553    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-socket\") pod \"3d43d33e-3b63-4c69-ae89-480cef886400\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615606    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istiod-ca-cert\") pod \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615737    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-certs\") pod \"3d43d33e-3b63-4c69-ae89-480cef886400\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615787    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-podinfo\") pod \"cf435d77-26a8-436f-b3fc-b0a1235bd325\" (UID: \"cf435d77-26a8-436f-b3fc-b0a1235bd325\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615821    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-socket\") pod \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615890    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-socket\") pod \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.615957    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-credential-socket\") pod \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\" (UID: \"e8c4b040-6e11-4236-b7f2-9438685e5ba3\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.616009    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d43d33e-3b63-4c69-ae89-480cef886400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d43d33e-3b63-4c69-ae89-480cef886400-istiod-ca-cert\") pod \"3d43d33e-3b63-4c69-ae89-480cef886400\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.616047    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-data\") pod \"3d43d33e-3b63-4c69-ae89-480cef886400\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.616089    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-envoy\") pod \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\" (UID: \"dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.616132    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-token\" (UniqueName: \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-token\") pod \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\" (UID: \"256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b\") "
Nov 02 01:36:48.618343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.616168    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-istio-token\" (UniqueName: \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-istio-token\") pod \"3d43d33e-3b63-4c69-ae89-480cef886400\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") "
Nov 02 01:36:48.622306 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.622243    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-token" pod "cf435d77-26a8-436f-b3fc-b0a1235bd325" (UID: "cf435d77-26a8-436f-b3fc-b0a1235bd325"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.622826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.622802    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-kube-api-access-n4tjl" pod "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" (UID: "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e"). InnerVolumeSpecName "kube-api-access-n4tjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.623009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.622835    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-kube-api-access-ws8lk" pod "cf435d77-26a8-436f-b3fc-b0a1235bd325" (UID: "cf435d77-26a8-436f-b3fc-b0a1235bd325"). InnerVolumeSpecName "kube-api-access-ws8lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.623133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.622954    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-envoy" pod "3d43d33e-3b63-4c69-ae89-480cef886400" (UID: "3d43d33e-3b63-4c69-ae89-480cef886400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.623720 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.623693    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-socket" pod "3d43d33e-3b63-4c69-ae89-480cef886400" (UID: "3d43d33e-3b63-4c69-ae89-480cef886400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.624156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.624132    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-certs" pod "3d43d33e-3b63-4c69-ae89-480cef886400" (UID: "3d43d33e-3b63-4c69-ae89-480cef886400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.630481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.625220    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istiod-ca-cert" pod "e8c4b040-6e11-4236-b7f2-9438685e5ba3" (UID: "e8c4b040-6e11-4236-b7f2-9438685e5ba3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:36:48.630618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.626549    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-kube-api-access-c5nkp\" (UniqueName: \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-kube-api-access-c5nkp\") pod \"3d43d33e-3b63-4c69-ae89-480cef886400\" (UID: \"3d43d33e-3b63-4c69-ae89-480cef886400\") "
Nov 02 01:36:48.650528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.632551    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8lk\" (UniqueName: \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-kube-api-access-ws8lk\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.671387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.671353    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.671645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.671624    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4tjl\" (UniqueName: \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-kube-api-access-n4tjl\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.671752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.671739    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.671871 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.624456    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:36:48.672479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.651253    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-podinfo" pod "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" (UID: "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:36:48.681389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.676017    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:36:48.696839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.696809    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:36:48.723874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.722816    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-kube-api-access-vh7p4" pod "e8c4b040-6e11-4236-b7f2-9438685e5ba3" (UID: "e8c4b040-6e11-4236-b7f2-9438685e5ba3"). InnerVolumeSpecName "kube-api-access-vh7p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.723874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.723593    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-podinfo" pod "e8c4b040-6e11-4236-b7f2-9438685e5ba3" (UID: "e8c4b040-6e11-4236-b7f2-9438685e5ba3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:36:48.723874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.723816    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-envoy" pod "e8c4b040-6e11-4236-b7f2-9438685e5ba3" (UID: "e8c4b040-6e11-4236-b7f2-9438685e5ba3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:48.724755    4851 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/cf435d77-26a8-436f-b3fc-b0a1235bd325/volumes/kubernetes.io~downward-api/istio-podinfo
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.724778    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-podinfo" pod "cf435d77-26a8-436f-b3fc-b0a1235bd325" (UID: "cf435d77-26a8-436f-b3fc-b0a1235bd325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.724876    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d43d33e-3b63-4c69-ae89-480cef886400-istiod-ca-cert" pod "3d43d33e-3b63-4c69-ae89-480cef886400" (UID: "3d43d33e-3b63-4c69-ae89-480cef886400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.724994    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-socket" pod "e8c4b040-6e11-4236-b7f2-9438685e5ba3" (UID: "e8c4b040-6e11-4236-b7f2-9438685e5ba3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.725325    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-credential-socket" pod "e8c4b040-6e11-4236-b7f2-9438685e5ba3" (UID: "e8c4b040-6e11-4236-b7f2-9438685e5ba3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.725366    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-socket" pod "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" (UID: "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.725572    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-credential-socket" pod "3d43d33e-3b63-4c69-ae89-480cef886400" (UID: "3d43d33e-3b63-4c69-ae89-480cef886400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.725732    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-credential-socket" pod "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" (UID: "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.725887    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-certs" pod "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" (UID: "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.728611    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-data" pod "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" (UID: "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.729113    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-data" pod "e8c4b040-6e11-4236-b7f2-9438685e5ba3" (UID: "e8c4b040-6e11-4236-b7f2-9438685e5ba3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.730100    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istiod-ca-cert" pod "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" (UID: "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:48.730157    4851 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e/volumes/kubernetes.io~empty-dir/istio-envoy
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.730167    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-envoy" pod "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" (UID: "dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.731390    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-data" pod "3d43d33e-3b63-4c69-ae89-480cef886400" (UID: "3d43d33e-3b63-4c69-ae89-480cef886400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.731523    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-kube-api-access-c5nkp" pod "3d43d33e-3b63-4c69-ae89-480cef886400" (UID: "3d43d33e-3b63-4c69-ae89-480cef886400"). InnerVolumeSpecName "kube-api-access-c5nkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.731584    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-envoy" pod "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" (UID: "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.732859    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d43d33e-3b63-4c69-ae89-480cef886400-istio-podinfo" pod "3d43d33e-3b63-4c69-ae89-480cef886400" (UID: "3d43d33e-3b63-4c69-ae89-480cef886400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.671853    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.748723    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.748744    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.748763    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.748778    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.767600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.750689    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-token" pod "e8c4b040-6e11-4236-b7f2-9438685e5ba3" (UID: "e8c4b040-6e11-4236-b7f2-9438685e5ba3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.798315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.795367    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-kube-api-access-8kj4d" pod "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" (UID: "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b"). InnerVolumeSpecName "kube-api-access-8kj4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.811654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.811083    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-token" pod "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" (UID: "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.816646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.816606    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-certs" pod "e8c4b040-6e11-4236-b7f2-9438685e5ba3" (UID: "e8c4b040-6e11-4236-b7f2-9438685e5ba3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.822263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.822219    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-podinfo" pod "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" (UID: "256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:36:48.825076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.825035    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-istio-token" pod "3d43d33e-3b63-4c69-ae89-480cef886400" (UID: "3d43d33e-3b63-4c69-ae89-480cef886400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.852326    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-socket\") pod \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.852448    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-credential-socket\") pod \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.852498    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-certs\") pod \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.852541    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c4b872-5d76-4900-b4ea-09331641a617-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c4b872-5d76-4900-b4ea-09331641a617-plugins-config-volume\") pod \"10c4b872-5d76-4900-b4ea-09331641a617\" (UID: \"10c4b872-5d76-4900-b4ea-09331641a617\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.852581    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-socket\") pod \"dd465d06-c445-4b09-a15f-5d97797c850c\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.860492    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-data\") pod \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.860559    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-credential-socket\") pod \"dd465d06-c445-4b09-a15f-5d97797c850c\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.860687    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c4b872-5d76-4900-b4ea-09331641a617-kube-api-access-knfkz\" (UniqueName: \"kubernetes.io/projected/10c4b872-5d76-4900-b4ea-09331641a617-kube-api-access-knfkz\") pod \"10c4b872-5d76-4900-b4ea-09331641a617\" (UID: \"10c4b872-5d76-4900-b4ea-09331641a617\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.860725    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-certs\") pod \"dd465d06-c445-4b09-a15f-5d97797c850c\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.860811    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-envoy\") pod \"dd465d06-c445-4b09-a15f-5d97797c850c\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.860911    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd465d06-c445-4b09-a15f-5d97797c850c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd465d06-c445-4b09-a15f-5d97797c850c-istiod-ca-cert\") pod \"dd465d06-c445-4b09-a15f-5d97797c850c\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.860952    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-istio-token\" (UniqueName: \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-istio-token\") pod \"dd465d06-c445-4b09-a15f-5d97797c850c\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.860988    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-token\" (UniqueName: \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-token\") pod \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.861093    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-kube-api-access-q58p9\" (UniqueName: \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-kube-api-access-q58p9\") pod \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.861134    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-kube-api-access-p8twd\" (UniqueName: \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-kube-api-access-p8twd\") pod \"dd465d06-c445-4b09-a15f-5d97797c850c\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.861195    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd465d06-c445-4b09-a15f-5d97797c850c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd465d06-c445-4b09-a15f-5d97797c850c-istio-podinfo\") pod \"dd465d06-c445-4b09-a15f-5d97797c850c\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") "
Nov 02 01:36:48.861311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.861224    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-podinfo\") pod \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") "
Nov 02 01:36:48.882866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.879364    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-certs" pod "dd465d06-c445-4b09-a15f-5d97797c850c" (UID: "dd465d06-c445-4b09-a15f-5d97797c850c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.882866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.879621    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-socket" pod "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" (UID: "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.882866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.879837    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-credential-socket" pod "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" (UID: "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.882866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.880002    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-certs" pod "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" (UID: "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.882866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.880324    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c4b872-5d76-4900-b4ea-09331641a617-plugins-config-volume" pod "10c4b872-5d76-4900-b4ea-09331641a617" (UID: "10c4b872-5d76-4900-b4ea-09331641a617"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:36:48.882866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.880639    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-socket" pod "dd465d06-c445-4b09-a15f-5d97797c850c" (UID: "dd465d06-c445-4b09-a15f-5d97797c850c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.882866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.880871    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-data" pod "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" (UID: "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.882866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.881064    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-credential-socket" pod "dd465d06-c445-4b09-a15f-5d97797c850c" (UID: "dd465d06-c445-4b09-a15f-5d97797c850c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.901620    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istiod-ca-cert\") pod \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") "
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.901670    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-data\") pod \"dd465d06-c445-4b09-a15f-5d97797c850c\" (UID: \"dd465d06-c445-4b09-a15f-5d97797c850c\") "
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.901714    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-envoy\") pod \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\" (UID: \"27f6f0dd-c9c5-4b89-a839-60d5d1c5a524\") "
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902726    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d43d33e-3b63-4c69-ae89-480cef886400-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902761    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902775    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902790    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902803    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902818    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf435d77-26a8-436f-b3fc-b0a1235bd325-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902831    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902844    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902857    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902869    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d43d33e-3b63-4c69-ae89-480cef886400-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902881    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902893    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902906    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902919    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902932    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5nkp\" (UniqueName: \"kubernetes.io/projected/3d43d33e-3b63-4c69-ae89-480cef886400-kube-api-access-c5nkp\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902947    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902959    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh7p4\" (UniqueName: \"kubernetes.io/projected/e8c4b040-6e11-4236-b7f2-9438685e5ba3-kube-api-access-vh7p4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902972    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kj4d\" (UniqueName: \"kubernetes.io/projected/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-kube-api-access-8kj4d\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902984    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.902996    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903009    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d43d33e-3b63-4c69-ae89-480cef886400-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903024    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903038    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903057    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903070    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903083    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c4b872-5d76-4900-b4ea-09331641a617-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903095    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903107    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903120    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903134    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903147    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903161    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4b040-6e11-4236-b7f2-9438685e5ba3-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903175    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.910572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.903189    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:48.918121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.914900    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd465d06-c445-4b09-a15f-5d97797c850c-istiod-ca-cert" pod "dd465d06-c445-4b09-a15f-5d97797c850c" (UID: "dd465d06-c445-4b09-a15f-5d97797c850c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:36:48.918121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.917748    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-data" pod "dd465d06-c445-4b09-a15f-5d97797c850c" (UID: "dd465d06-c445-4b09-a15f-5d97797c850c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.918517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.918460    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istiod-ca-cert" pod "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" (UID: "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:36:48.919024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.918988    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c4b872-5d76-4900-b4ea-09331641a617-kube-api-access-knfkz" pod "10c4b872-5d76-4900-b4ea-09331641a617" (UID: "10c4b872-5d76-4900-b4ea-09331641a617"). InnerVolumeSpecName "kube-api-access-knfkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.920063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.920013    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-token" pod "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" (UID: "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.922241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.921677    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-envoy" pod "dd465d06-c445-4b09-a15f-5d97797c850c" (UID: "dd465d06-c445-4b09-a15f-5d97797c850c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:48.922241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.922012    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-podinfo" pod "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" (UID: "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:36:48.922241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.922026    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd465d06-c445-4b09-a15f-5d97797c850c-istio-podinfo" pod "dd465d06-c445-4b09-a15f-5d97797c850c" (UID: "dd465d06-c445-4b09-a15f-5d97797c850c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:36:48.922792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.922753    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-kube-api-access-q58p9" pod "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" (UID: "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524"). InnerVolumeSpecName "kube-api-access-q58p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.923074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.923029    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-istio-token" pod "dd465d06-c445-4b09-a15f-5d97797c850c" (UID: "dd465d06-c445-4b09-a15f-5d97797c850c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.923419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.923263    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-kube-api-access-p8twd" pod "dd465d06-c445-4b09-a15f-5d97797c850c" (UID: "dd465d06-c445-4b09-a15f-5d97797c850c"). InnerVolumeSpecName "kube-api-access-p8twd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:36:48.923619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:48.923599    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-envoy" pod "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" (UID: "27f6f0dd-c9c5-4b89-a839-60d5d1c5a524"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:36:49.225589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.101956    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:49.225589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.201077    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:36:49.225589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.201135    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.316953    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knfkz\" (UniqueName: \"kubernetes.io/projected/10c4b872-5d76-4900-b4ea-09331641a617-kube-api-access-knfkz\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.316989    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.317004    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd465d06-c445-4b09-a15f-5d97797c850c-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.317018    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.317032    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.317045    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q58p9\" (UniqueName: \"kubernetes.io/projected/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-kube-api-access-q58p9\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.317059    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8twd\" (UniqueName: \"kubernetes.io/projected/dd465d06-c445-4b09-a15f-5d97797c850c-kube-api-access-p8twd\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.317072    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd465d06-c445-4b09-a15f-5d97797c850c-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.317085    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.317097    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.317110    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd465d06-c445-4b09-a15f-5d97797c850c-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.353570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.317122    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:36:49.422792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.421190    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk" event={"ID":"10c4b872-5d76-4900-b4ea-09331641a617","Type":"ContainerDied","Data":"543fa9c3264df321f19c177ffb8b6ebdad137ab7ec05c93560e5aa3505947b26"}
Nov 02 01:36:49.422792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.421269    4851 scope.go:122] "RemoveContainer" containerID="3a5fdf2bd6a64649919a647a1f3d18c7b9bd7ae7e508581d21a8e39de92ee9a7"
Nov 02 01:36:49.422792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.421436    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"
Nov 02 01:36:49.445327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.438054    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"]
Nov 02 01:36:49.493929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.493749    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"
Nov 02 01:36:49.494099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.494071    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"
Nov 02 01:36:49.494454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.494428    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"
Nov 02 01:36:49.495658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.494642    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"
Nov 02 01:36:49.495658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.494813    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"
Nov 02 01:36:49.495658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.495248    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"
Nov 02 01:36:49.498708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.498686    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-2m4f6"]
Nov 02 01:36:49.511555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.511532    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"]
Nov 02 01:36:49.518319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.518278    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50537f4f-5d749d4b6c-95chk"]
Nov 02 01:36:49.598042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.598008    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"]
Nov 02 01:36:49.603580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.603552    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-kcg25"]
Nov 02 01:36:49.610277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.610246    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"]
Nov 02 01:36:49.614261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.614238    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-5zlkd"]
Nov 02 01:36:49.623932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.623912    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"]
Nov 02 01:36:49.685313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.681374    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-2kqjd"]
Nov 02 01:36:49.727057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.724554    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"]
Nov 02 01:36:49.728140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.728091    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-8jmls"]
Nov 02 01:36:49.870363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.870327    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"]
Nov 02 01:36:49.914312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.909378    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-phhf6"]
Nov 02 01:36:49.915418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.915361    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"]
Nov 02 01:36:49.922970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:49.921844    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ptf92"]
Nov 02 01:36:50.453330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:50.453275    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:50.895976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:50.895890    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c4b872-5d76-4900-b4ea-09331641a617" path="/var/lib/kubelet/pods/10c4b872-5d76-4900-b4ea-09331641a617/volumes"
Nov 02 01:36:50.896560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:50.896490    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" path="/var/lib/kubelet/pods/256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b/volumes"
Nov 02 01:36:50.897060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:50.897043    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" path="/var/lib/kubelet/pods/27f6f0dd-c9c5-4b89-a839-60d5d1c5a524/volumes"
Nov 02 01:36:50.897892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:50.897876    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" path="/var/lib/kubelet/pods/3d43d33e-3b63-4c69-ae89-480cef886400/volumes"
Nov 02 01:36:50.898378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:50.898362    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" path="/var/lib/kubelet/pods/cf435d77-26a8-436f-b3fc-b0a1235bd325/volumes"
Nov 02 01:36:50.898822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:50.898808    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" path="/var/lib/kubelet/pods/dd465d06-c445-4b09-a15f-5d97797c850c/volumes"
Nov 02 01:36:50.899548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:50.899532    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" path="/var/lib/kubelet/pods/dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e/volumes"
Nov 02 01:36:50.899965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:50.899952    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" path="/var/lib/kubelet/pods/e8c4b040-6e11-4236-b7f2-9438685e5ba3/volumes"
Nov 02 01:36:51.549961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:51.549925    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:51.791922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:51.791885    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:52.005318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.004416    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"]
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005523    4851 cpu_manager.go:414] "containerMap: removing container" podUID="10c4b872-5d76-4900-b4ea-09331641a617" containerName="payload-pre-processing"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005542    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c4b872-5d76-4900-b4ea-09331641a617" containerName="payload-pre-processing"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005562    4851 cpu_manager.go:414] "containerMap: removing container" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005571    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005614    4851 cpu_manager.go:414] "containerMap: removing container" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005622    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005641    4851 cpu_manager.go:414] "containerMap: removing container" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005648    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005661    4851 cpu_manager.go:414] "containerMap: removing container" podUID="23d4e564-ddd9-4415-9823-6e39b5c314f3" containerName="revoke-keys"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005668    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d4e564-ddd9-4415-9823-6e39b5c314f3" containerName="revoke-keys"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005681    4851 cpu_manager.go:414] "containerMap: removing container" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005688    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005712    4851 cpu_manager.go:414] "containerMap: removing container" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005719    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005734    4851 cpu_manager.go:414] "containerMap: removing container" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005740    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005767    4851 cpu_manager.go:414] "containerMap: removing container" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy"
Nov 02 01:36:52.005802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005774    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy"
Nov 02 01:36:52.006671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005952    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="256d8fe9-92f6-42ed-9a7b-e7f80d55ae2b" containerName="istio-proxy"
Nov 02 01:36:52.006671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005964    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d4e564-ddd9-4415-9823-6e39b5c314f3" containerName="revoke-keys"
Nov 02 01:36:52.006671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005979    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f6f0dd-c9c5-4b89-a839-60d5d1c5a524" containerName="istio-proxy"
Nov 02 01:36:52.006671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.005994    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c4b040-6e11-4236-b7f2-9438685e5ba3" containerName="istio-proxy"
Nov 02 01:36:52.006671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.006003    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d43d33e-3b63-4c69-ae89-480cef886400" containerName="istio-proxy"
Nov 02 01:36:52.006671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.006016    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd465d06-c445-4b09-a15f-5d97797c850c" containerName="istio-proxy"
Nov 02 01:36:52.006671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.006027    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c4b872-5d76-4900-b4ea-09331641a617" containerName="payload-pre-processing"
Nov 02 01:36:52.006671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.006038    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf435d77-26a8-436f-b3fc-b0a1235bd325" containerName="istio-proxy"
Nov 02 01:36:52.006671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.006050    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee1ace2-7f9e-4785-b1a0-c093dc0d0b1e" containerName="istio-proxy"
Nov 02 01:36:52.030378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.010580    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.030378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.019500    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"]
Nov 02 01:36:52.030378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.030202    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bf1e55-openshift-default-dockercfg-w8kcb\""
Nov 02 01:36:52.055366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.052966    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:52.143814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.143783    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.144007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.143897    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.144007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.143958    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.180316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.144027    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.180316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.144259    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.180316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.144348    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx4wj\" (UniqueName: \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-kube-api-access-xx4wj\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.180316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.144451    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.180316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.144493    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.180316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.144549    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.268455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.268357    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.286687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.269157    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.287677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.287629    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.287820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.286605    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.287982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.287967    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.293324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.288483    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.301554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.301527    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.305320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.302157    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx4wj\" (UniqueName: \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-kube-api-access-xx4wj\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.416890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.416856    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.417565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.417519    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.418213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.418190    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.418538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.418509    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.418615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.417438    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.418615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.407209    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx4wj\" (UniqueName: \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-kube-api-access-xx4wj\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.418615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.294146    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.418615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.377909    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:52.464228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.464192    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.472655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.466355    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.504402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.503629    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.663092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.663065    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:52.708622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:52.708590    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:53.171028 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:53.170995    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"]
Nov 02 01:36:53.173190 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:53.173155    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae956ce_3eb3_44b7_a7c1_487ec5a5d21b.slice/crio-db53cebf2866c0f98c5a9a345814809e86996420f32f60f5cec5a747da80f1af WatchSource:0}: Error finding container db53cebf2866c0f98c5a9a345814809e86996420f32f60f5cec5a747da80f1af: Status 404 returned error can't find the container with id db53cebf2866c0f98c5a9a345814809e86996420f32f60f5cec5a747da80f1af
Nov 02 01:36:53.176636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:53.176598    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:53.176757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:53.176699    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:53.176757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:53.176743    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:53.251453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:53.251420    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:53.601238 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:53.601143    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:53.625191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:53.625127    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" event={"ID":"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b","Type":"ContainerStarted","Data":"f1769e89b2b8bab680c9f8909458a125a26c2926c597f4cb936a09869cf1c68c"}
Nov 02 01:36:53.625405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:53.625260    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" event={"ID":"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b","Type":"ContainerStarted","Data":"db53cebf2866c0f98c5a9a345814809e86996420f32f60f5cec5a747da80f1af"}
Nov 02 01:36:53.685316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:53.684707    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:53.725505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:53.725440    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podStartSLOduration=2.725420835 podStartE2EDuration="2.725420835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:53.650058835 +0000 UTC m=+1318.954991661" watchObservedRunningTime="2026-11-02 01:36:53.725420835 +0000 UTC m=+1319.030353661"
Nov 02 01:36:54.671657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:54.671257    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:54.690681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:54.690652    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:54.941638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:54.941545    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:55.159842 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:55.159802    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:55.663490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:55.663445    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:55.664519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:55.664496    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:36:55.987983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:55.987897    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:56.058599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.058568    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"]
Nov 02 01:36:56.166307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.166256    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"]
Nov 02 01:36:56.166513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.166396    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.175322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.175278    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-addad9\"/\"kube-root-ca.crt\""
Nov 02 01:36:56.175544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.175421    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-addad9\"/\"openshift-service-ca.crt\""
Nov 02 01:36:56.175544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.175463    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-addad9\"/\"default-dockercfg-tmf9b\""
Nov 02 01:36:56.176277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.176254    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-addad9\"/\"test-model-addad9-kserve-self-signed-certs\""
Nov 02 01:36:56.200946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.200903    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:56.220725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.220685    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-home\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.220725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.220717    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-dshm\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.220976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.220737    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-tmp-dir\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.220976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.220893    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1f2479-6898-45d6-8296-c68deb4b98bf-tls-certs\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.221097 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.221012    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8kx\" (UniqueName: \"kubernetes.io/projected/6e1f2479-6898-45d6-8296-c68deb4b98bf-kube-api-access-kd8kx\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.221154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.221140    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-model-cache\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.333558    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-home\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.333608    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-dshm\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.333632    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-tmp-dir\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.333675    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1f2479-6898-45d6-8296-c68deb4b98bf-tls-certs\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.333754    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8kx\" (UniqueName: \"kubernetes.io/projected/6e1f2479-6898-45d6-8296-c68deb4b98bf-kube-api-access-kd8kx\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.333875    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-model-cache\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.334415    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-model-cache\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.334644    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-home\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.343010    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-tmp-dir\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.382803    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1f2479-6898-45d6-8296-c68deb4b98bf-tls-certs\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.420433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.402139    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-dshm\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.471314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.443264    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8kx\" (UniqueName: \"kubernetes.io/projected/6e1f2479-6898-45d6-8296-c68deb4b98bf-kube-api-access-kd8kx\") pod \"test-model-addad9-kserve-cbcb4cd8f-l5kkp\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") " pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:56.604314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:56.603611    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:57.000663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.000631    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"]
Nov 02 01:36:57.002752 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:57.002711    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e1f2479_6898_45d6_8296_c68deb4b98bf.slice/crio-9e0a432fa8b8305be346c12f1cd369e2ca318c89d03122aea42da85588de5866 WatchSource:0}: Error finding container 9e0a432fa8b8305be346c12f1cd369e2ca318c89d03122aea42da85588de5866: Status 404 returned error can't find the container with id 9e0a432fa8b8305be346c12f1cd369e2ca318c89d03122aea42da85588de5866
Nov 02 01:36:57.059885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.059845    4851 scope.go:122] "RemoveContainer" containerID="4c8e9f7b8a8a20b0c5620f8889e78aa1a501283ddbace726cd0e244dd46aedb0"
Nov 02 01:36:57.125863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.125821    4851 scope.go:122] "RemoveContainer" containerID="dd6b7ca333c575bc4734fe89fb813194f99b84d8ef0586cdf06e8d46fec2a338"
Nov 02 01:36:57.182400 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.182363    4851 scope.go:122] "RemoveContainer" containerID="f8767c1a789b9eb72fc6cdc433e916972d0741e87d8916105f08d46fdd93bdce"
Nov 02 01:36:57.216883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.216853    4851 scope.go:122] "RemoveContainer" containerID="1ca66c6a16d173d4844902705250627ccf6eb79c453d43f0bf651bd750ef7ab0"
Nov 02 01:36:57.242257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.242196    4851 scope.go:122] "RemoveContainer" containerID="c2ba7d93b5c2de35816f618cb8d4530995e4d7f70f3e37c03bc92b844afe9c5c"
Nov 02 01:36:57.279815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.279790    4851 scope.go:122] "RemoveContainer" containerID="62529d00135af65a8690c4336ceb0e4a3ff57ff34f372817876611ff3cb7aae1"
Nov 02 01:36:57.303487 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.303433    4851 scope.go:122] "RemoveContainer" containerID="530fa39b156027983bdd36ab12fa28256897addba275241680b867749d73f78c"
Nov 02 01:36:57.324363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.324337    4851 scope.go:122] "RemoveContainer" containerID="36888737e7a47a361d80260372b8a32bdf175e7e8acb5843198c7910302780d5"
Nov 02 01:36:57.366119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.366099    4851 scope.go:122] "RemoveContainer" containerID="ad6a3825f1f3291edcdd94abfd5bf160d89757355ae8b500e374e08c80c37d69"
Nov 02 01:36:57.390768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.390719    4851 scope.go:122] "RemoveContainer" containerID="cd0b18a5667aad25db9afd9b677900e4430d2363921dde4c082e42b5ff77d468"
Nov 02 01:36:57.431433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.418767    4851 scope.go:122] "RemoveContainer" containerID="763f1dc8bf4d638ff7e266004c26036551a8cda3bf4a54bbcf547b0a0dca0ffa"
Nov 02 01:36:57.470126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.470092    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:57.770580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.770442    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:57.798949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.797761    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" event={"ID":"6e1f2479-6898-45d6-8296-c68deb4b98bf","Type":"ContainerStarted","Data":"fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b"}
Nov 02 01:36:57.798949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.797812    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" event={"ID":"6e1f2479-6898-45d6-8296-c68deb4b98bf","Type":"ContainerStarted","Data":"9e0a432fa8b8305be346c12f1cd369e2ca318c89d03122aea42da85588de5866"}
Nov 02 01:36:57.837929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.837892    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"]
Nov 02 01:36:57.850430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.843748    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podStartSLOduration=1.8437239600000002 podStartE2EDuration="1.84372396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:57.821797647 +0000 UTC m=+1323.126730474" watchObservedRunningTime="2026-11-02 01:36:57.84372396 +0000 UTC m=+1323.148656788"
Nov 02 01:36:57.860705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.859981    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"]
Nov 02 01:36:57.861470 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.861419    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:36:57.871331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.865444    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"]
Nov 02 01:36:57.874606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.873504    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bf1e55\""
Nov 02 01:36:57.874606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.873639    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bf1e55-dockercfg-xhgrn\""
Nov 02 01:36:57.890550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.890519    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"]
Nov 02 01:36:57.890752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.890646    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:36:57.920006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.919276    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khlqm\" (UniqueName: \"kubernetes.io/projected/5eafdd80-7407-40d6-9579-cedb558b6411-kube-api-access-khlqm\") pod \"payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz\" (UID: \"5eafdd80-7407-40d6-9579-cedb558b6411\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:36:57.923370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:57.922622    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eafdd80-7407-40d6-9579-cedb558b6411-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz\" (UID: \"5eafdd80-7407-40d6-9579-cedb558b6411\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:36:58.061680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.040229    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:58.074317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.062852    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m\" (UID: \"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:36:58.074317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.062991    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khlqm\" (UniqueName: \"kubernetes.io/projected/5eafdd80-7407-40d6-9579-cedb558b6411-kube-api-access-khlqm\") pod \"payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz\" (UID: \"5eafdd80-7407-40d6-9579-cedb558b6411\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:36:58.074317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.063216    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q74b5\" (UniqueName: \"kubernetes.io/projected/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-kube-api-access-q74b5\") pod \"payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m\" (UID: \"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:36:58.074317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.063346    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eafdd80-7407-40d6-9579-cedb558b6411-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz\" (UID: \"5eafdd80-7407-40d6-9579-cedb558b6411\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:36:58.074317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.064374    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eafdd80-7407-40d6-9579-cedb558b6411-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz\" (UID: \"5eafdd80-7407-40d6-9579-cedb558b6411\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:36:58.092377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.087530    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"]
Nov 02 01:36:58.155316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.145754    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.155316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.150414    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"]
Nov 02 01:36:58.166076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.165863    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q74b5\" (UniqueName: \"kubernetes.io/projected/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-kube-api-access-q74b5\") pod \"payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m\" (UID: \"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:36:58.248889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.166402    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m\" (UID: \"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:36:58.248889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.167638    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m\" (UID: \"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:36:58.248889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.243525    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khlqm\" (UniqueName: \"kubernetes.io/projected/5eafdd80-7407-40d6-9579-cedb558b6411-kube-api-access-khlqm\") pod \"payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz\" (UID: \"5eafdd80-7407-40d6-9579-cedb558b6411\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:36:58.277699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.268840    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.277699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.268897    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.277699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.268975    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.277699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.269033    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.277699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.269114    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.277699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.274173    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.277699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.274403    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twhzz\" (UniqueName: \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-kube-api-access-twhzz\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.277699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.274506    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.277699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.274556    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.343900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.290947    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q74b5\" (UniqueName: \"kubernetes.io/projected/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-kube-api-access-q74b5\") pod \"payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m\" (UID: \"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:36:58.411031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.380533    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.411031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.380619    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.411031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.380707    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.411031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.380752    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.411031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.380838    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.411031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.380902    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.411031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.380989    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.411031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.381059    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.411031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.381205    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twhzz\" (UniqueName: \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-kube-api-access-twhzz\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.427651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.427522    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.449538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.428725    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.449538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.428793    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.449538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.428982    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.449538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.429111    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.455849 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.455818    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.481015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.474731    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twhzz\" (UniqueName: \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-kube-api-access-twhzz\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.481015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.476607    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.481015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.477134    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.508324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.508274    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:36:58.551320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.551186    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:36:58.605257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.605189    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:36:58.646765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.646064    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:36:58.884680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.884650    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"]
Nov 02 01:36:58.889585 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:58.889553    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eafdd80_7407_40d6_9579_cedb558b6411.slice/crio-3fd8d15f4bcfe32ed06895e7138a854ae4e350221a96d838fd1e3739e0bb88e0 WatchSource:0}: Error finding container 3fd8d15f4bcfe32ed06895e7138a854ae4e350221a96d838fd1e3739e0bb88e0: Status 404 returned error can't find the container with id 3fd8d15f4bcfe32ed06895e7138a854ae4e350221a96d838fd1e3739e0bb88e0
Nov 02 01:36:58.912418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.912386    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:36:58.932837 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:58.932812    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"]
Nov 02 01:36:59.065985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.065949    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"]
Nov 02 01:36:59.105095 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:36:59.102145    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb94e95_465c_49e8_9ad5_dc74fa3283bc.slice/crio-0d280970273f9dff1c40e4228ae00df818534be579b30de5925581eaec2a23c2 WatchSource:0}: Error finding container 0d280970273f9dff1c40e4228ae00df818534be579b30de5925581eaec2a23c2: Status 404 returned error can't find the container with id 0d280970273f9dff1c40e4228ae00df818534be579b30de5925581eaec2a23c2
Nov 02 01:36:59.120546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.120106    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:59.120668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.120267    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:59.120881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.120835    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:59.826980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.826941    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz" event={"ID":"5eafdd80-7407-40d6-9579-cedb558b6411","Type":"ContainerStarted","Data":"679ca1dd80b9558dc896e5342c5463cf5ee3d884fe796199e92f43a1e5436601"}
Nov 02 01:36:59.830905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.827054    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:36:59.830905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.827075    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz" event={"ID":"5eafdd80-7407-40d6-9579-cedb558b6411","Type":"ContainerStarted","Data":"3fd8d15f4bcfe32ed06895e7138a854ae4e350221a96d838fd1e3739e0bb88e0"}
Nov 02 01:36:59.830905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.828722    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" event={"ID":"7cb94e95-465c-49e8-9ad5-dc74fa3283bc","Type":"ContainerStarted","Data":"2d458e5ff27410e3e425aef91593ccbeb56a2ef8d87aee2aebbe565781de2972"}
Nov 02 01:36:59.830905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.828759    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" event={"ID":"7cb94e95-465c-49e8-9ad5-dc74fa3283bc","Type":"ContainerStarted","Data":"0d280970273f9dff1c40e4228ae00df818534be579b30de5925581eaec2a23c2"}
Nov 02 01:36:59.830905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.830191    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m" event={"ID":"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06","Type":"ContainerStarted","Data":"40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0"}
Nov 02 01:36:59.830905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.830233    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:36:59.830905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.830250    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m" event={"ID":"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06","Type":"ContainerStarted","Data":"1a29a457590a2de0e9d3f3a58283466809bed682dc11e53cc2162e4167eacba3"}
Nov 02 01:36:59.853307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.853236    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz" podStartSLOduration=2.853219347 podStartE2EDuration="2.853219347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:59.8449145 +0000 UTC m=+1325.149847350" watchObservedRunningTime="2026-11-02 01:36:59.853219347 +0000 UTC m=+1325.158152174"
Nov 02 01:36:59.865982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.865933    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m" podStartSLOduration=2.865918037 podStartE2EDuration="2.865918037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:59.862121899 +0000 UTC m=+1325.167054728" watchObservedRunningTime="2026-11-02 01:36:59.865918037 +0000 UTC m=+1325.170850863"
Nov 02 01:36:59.887450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:36:59.887395    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" podStartSLOduration=1.887376519 podStartE2EDuration="1.887376519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:59.88400413 +0000 UTC m=+1325.188936956" watchObservedRunningTime="2026-11-02 01:36:59.887376519 +0000 UTC m=+1325.192309343"
Nov 02 01:37:00.013435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.013399    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:00.658371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.657609    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:37:00.659209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.659175    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 01:37:00.659379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.659256    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 01:37:00.867137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.867102    4851 generic.go:356] "Generic (PLEG): container finished" podID="da59dac5-1239-4ee9-b7af-f949e9891687" containerID="b9a724f46926be34a1037c594310b576b54f2ab11d96c0bb0170a59bb440c0fb" exitCode=0
Nov 02 01:37:00.868907 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.868879    4851 generic.go:356] "Generic (PLEG): container finished" podID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerID="2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41" exitCode=0
Nov 02 01:37:00.869038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.868883    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" event={"ID":"da59dac5-1239-4ee9-b7af-f949e9891687","Type":"ContainerDied","Data":"b9a724f46926be34a1037c594310b576b54f2ab11d96c0bb0170a59bb440c0fb"}
Nov 02 01:37:00.869038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.869010    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" event={"ID":"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa","Type":"ContainerDied","Data":"2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41"}
Nov 02 01:37:00.869764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.869749    4851 scope.go:122] "RemoveContainer" containerID="b9a724f46926be34a1037c594310b576b54f2ab11d96c0bb0170a59bb440c0fb"
Nov 02 01:37:00.870208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.870193    4851 scope.go:122] "RemoveContainer" containerID="2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41"
Nov 02 01:37:00.871533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.871503    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:00.871670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.871581    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:00.871670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.871632    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:00.871790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.871701    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:00.871790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.871762    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:00.871895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:00.871802    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:01.020825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.020783    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:01.326949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.326920    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"]
Nov 02 01:37:01.331223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.331200    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.368228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.349398    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0bbeda-openshift-default-dockercfg-8m89f\""
Nov 02 01:37:01.368228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.364656    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:37:01.373096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.373069    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"]
Nov 02 01:37:01.379551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.379520    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-envoy\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.379672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.379589    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-credential-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.379720 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.379680    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7rn\" (UniqueName: \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-kube-api-access-hm7rn\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.380011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.379991    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24da9590-78c7-472e-aa84-a64c8fda59a2-istiod-ca-cert\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.380190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.380171    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-token\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.380258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.380242    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.380381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.380344    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-podinfo\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.380483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.380450    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-certs\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.380548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.380496    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-data\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.481664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.481623    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.481853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.481675    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-podinfo\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.481853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.481716    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-certs\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.481853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.481746    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-data\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.481853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.481803    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-envoy\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.481853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.481825    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-credential-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.482156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.481868    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7rn\" (UniqueName: \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-kube-api-access-hm7rn\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.482156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.482020    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24da9590-78c7-472e-aa84-a64c8fda59a2-istiod-ca-cert\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.482156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.482102    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-token\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.482156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.482112    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.482437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.482222    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-certs\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.482506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.482442    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-data\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.482952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.482926    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24da9590-78c7-472e-aa84-a64c8fda59a2-istiod-ca-cert\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.482952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.482947    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-credential-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.485086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.485059    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-podinfo\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.486028 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.485406    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-envoy\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.495473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.495450    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7rn\" (UniqueName: \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-kube-api-access-hm7rn\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.495589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.495567    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-token\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.665846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.665492    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 01:37:01.665846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.665557    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 01:37:01.686015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.685984    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:01.907995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.888804    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" event={"ID":"da59dac5-1239-4ee9-b7af-f949e9891687","Type":"ContainerStarted","Data":"e4e735e6a235d02542919c015c294de6e7f33ceef0038a69157655bf471e4628"}
Nov 02 01:37:01.918930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:01.918749    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" event={"ID":"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa","Type":"ContainerStarted","Data":"ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32"}
Nov 02 01:37:02.102320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.100886    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"]
Nov 02 01:37:02.176360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.160090    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:02.211531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.195319    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:02.211531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.195423    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:02.211531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.195471    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:02.515432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.501354    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:37:02.638318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.627615    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 01:37:02.638318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.627666    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 01:37:02.662318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.646725    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 01:37:02.662318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.646795    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 01:37:02.694317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.693133    4851 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 01:37:02.694317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.693209    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 01:37:02.694317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:02.693528    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:37:03.116371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.115623    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" event={"ID":"24da9590-78c7-472e-aa84-a64c8fda59a2","Type":"ContainerStarted","Data":"98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074"}
Nov 02 01:37:03.116371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.115685    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" event={"ID":"24da9590-78c7-472e-aa84-a64c8fda59a2","Type":"ContainerStarted","Data":"2d5fd464cd3b002173a10b89f4718b933af404131b22fe218a8c07dbe0a3327c"}
Nov 02 01:37:03.256316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.230865    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podStartSLOduration=2.230845262 podStartE2EDuration="2.230845262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:03.150122759 +0000 UTC m=+1328.455055586" watchObservedRunningTime="2026-11-02 01:37:03.230845262 +0000 UTC m=+1328.535778093"
Nov 02 01:37:03.341314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.335472    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:03.448384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.447708    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 01:37:03.448384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.447767    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 01:37:03.649320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.648643    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 01:37:03.649320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.648702    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 01:37:03.696318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.696079    4851 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 02 01:37:03.696318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.696166    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 02 01:37:03.696888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.696586    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:03.704376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.696941    4851 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 01:37:03.704376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:03.696995    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 01:37:04.376320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:04.356062    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:04.445608 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:04.444664    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 01:37:04.445608 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:04.444767    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 01:37:04.986841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:04.957311    4851 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 02 01:37:04.986841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:04.957373    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 02 01:37:04.986841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:04.957835    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 01:37:04.986841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:04.957876    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 01:37:04.986841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:04.957951    4851 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 01:37:04.986841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:04.957970    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 01:37:05.184554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.184500    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:05.245415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.243990    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:37:05.267325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.245664    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:05.510131 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.358054    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:05.510131 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.358398    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:37:05.517121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.406674    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:05.522492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.491268    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 01:37:05.522765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.522715    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 01:37:05.546335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.545981    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:05.553870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.553826    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:05.621329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.602412    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:05.728745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.728702    4851 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 02 01:37:05.904370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.859342    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 02 01:37:05.904370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.754163    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:05.904370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.753635    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:05.904370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.763884    4851 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 01:37:05.904370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.901549    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 01:37:05.994444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:05.994026    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:06.018432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.018398    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:37:06.062118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.036383    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:06.085656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.037391    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 01:37:06.085977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.085924    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 01:37:06.187144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.187102    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:06.188076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.188051    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:37:06.315317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.286416    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:06.315317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.286948    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:37:06.317447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.317399    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:06.511866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.511777    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:37:06.519700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.512105    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:06.526491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.516012    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:06.526797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.519982    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:37:06.526909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.526874    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:37:06.526974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.526921    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:37:06.594315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.575894    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:06.602322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.595854    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:37:06.630319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.608844    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:37:06.630319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.613474    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:37:06.630319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.614007    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:37:06.630319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.623538    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:37:06.679317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.658639    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 01:37:06.693329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.684272    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 01:37:06.755760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.670106    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 01:37:06.769359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.756008    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 01:37:06.769638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.742392    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:06.769794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.688471    4851 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 02 01:37:06.769954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.769927    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 02 01:37:06.770083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.764622    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:37:06.770218 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.697360    4851 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 01:37:06.770363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.770343    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 01:37:06.849740 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.849700    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:06.879392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.879354    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:37:06.880066 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.879559    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:06.965319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:06.960030    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:37:07.010999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.010937    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:07.108923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.108828    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:37:07.190581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.190474    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:37:07.286766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.286715    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:07.302325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.290155    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:37:07.302325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.290239    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:37:07.302325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.290275    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:37:07.353898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.353871    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:37:07.527405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.520864    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:07.527756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.527720    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:37:07.538852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.535693    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:37:07.565555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.560824    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 01:37:07.565555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.561393    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 01:37:07.637324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.629398    4851 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:07.637324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.630651    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:07.698955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.698918    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:07.902915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.902818    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:37:07.905882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:07.903379    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:08.017321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.017260    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:08.040512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.040481    4851 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 01:37:08.040780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.040744    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 01:37:08.041445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.041424    4851 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 02 01:37:08.041630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.041610    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 02 01:37:08.048534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.048498    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:08.051971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.051948    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:37:08.118606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.118576    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:37:08.142815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.142772    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:08.231968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.231925    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:08.304318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.293907    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:08.310001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.306449    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:37:08.310001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.307840    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:37:08.494367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.489588    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 01:37:08.494367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.489651    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 01:37:08.520451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.509651    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:37:08.524311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.522838    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:08.555329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.550034    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:08.617762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.617721    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:08.625712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.625677    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:08.699189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.699133    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:08.712346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.700769    4851 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Nov 02 01:37:08.712677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.712629    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Nov 02 01:37:08.726093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.726051    4851 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 02 01:37:08.726467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.726384    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 02 01:37:08.803418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.803172    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:08.879524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.879177    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:08.911914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.909623    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:37:08.911914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:08.910154    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:09.072118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.050936    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:09.072118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.051907    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:09.176316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.170238    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:09.313468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.313424    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:09.318820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.318789    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:09.445673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.445639    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 01:37:09.445925 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.445896    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 01:37:09.450777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.450756    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:37:09.451242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.451224    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:37:09.529886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.529284    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:37:09.592938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.592850    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:37:09.810376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.758374    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:37:09.810376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.782766    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:37:09.880589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.876896    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:37:09.886325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.886276    4851 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 02 01:37:09.886623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.886584    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 02 01:37:09.887013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.886984    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:09.888846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.888748    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:37:09.888974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.888881    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:09.889848 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.889823    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"]
Nov 02 01:37:09.906800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.906629    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:37:09.909950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.909853    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:09.918495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.918468    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50537f4f-d9fbj"]
Nov 02 01:37:09.930007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.929983    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:37:09.932146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:09.932128    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:37:10.010017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.007519    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:37:10.014773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.014750    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:37:10.022056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.022013    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:37:10.026513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.026474    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:37:10.078402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.076247    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:37:10.475315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.471872    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:37:10.475315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.473156    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:37:10.475315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.475307    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:37:10.686732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.686696    4851 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 02 01:37:10.686947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.686762    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 02 01:37:10.779717 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.779624    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:10.957314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.954999    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d4e564-ddd9-4415-9823-6e39b5c314f3" path="/var/lib/kubelet/pods/23d4e564-ddd9-4415-9823-6e39b5c314f3/volumes"
Nov 02 01:37:10.957314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.955719    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:37:10.957314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:10.955781    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:37:11.475636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:11.475600    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:11.686660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:11.686621    4851 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 02 01:37:11.686870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:11.686673    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 02 01:37:12.285683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:12.285639    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:12.693014 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:12.692982    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:12.693614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:12.693547    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:12.694277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:12.694256    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:37:13.071085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:13.070984    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:13.825911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:13.825878    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:14.197887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:14.197857    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:14.424476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:14.424439    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:14.636318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:14.636222    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:15.382018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:15.381985    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:16.149189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:16.149153    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:16.617303 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:16.617263    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:37:16.629009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:16.628984    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:37:17.379505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:17.379472    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:17.599255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:17.599223    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:17.658578 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:37:17.658495    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:37:18.389069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:18.389030    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:19.351441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:19.351404    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:20.012934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:20.012903    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:20.779793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:20.779752    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:21.508960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:21.508909    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:21.953117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:21.953086    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:22.308860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.308749    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:22.523591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.523532    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:22.816835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.816801    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"]
Nov 02 01:37:22.823271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.823246    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:22.836062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.836033    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"]
Nov 02 01:37:22.837624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.837599    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f689bc-openshift-default-dockercfg-zlxb6\""
Nov 02 01:37:22.869304 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.869268    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:22.906640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.906595    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:22.958529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.958486    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:22.960044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.960017    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7p5q\" (UniqueName: \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-kube-api-access-g7p5q\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:22.993121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:22.993087    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.041636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.041599    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.042685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.042658    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.069480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.069393    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.069759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.069739    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.069886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.069870    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.176430    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.176546    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.176615    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.176646    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.176712    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.176763    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.176966    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7p5q\" (UniqueName: \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-kube-api-access-g7p5q\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.177004    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.177221    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.178120    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.184315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.178506    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.191312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.187178    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.191312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.187461    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.191312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.187693    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.245339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.223187    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.245339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.223720    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.268613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.268569    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7p5q\" (UniqueName: \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-kube-api-access-g7p5q\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.277312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.276766    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:23.281649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.281624    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.450319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.447614    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:23.505509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.497761    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:23.698636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.698600    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:23.904358 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:37:23.904327    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016dfa35_82cc_4ab0_a117_b5f6a9bb2ead.slice/crio-7d7539daaef63570cb9fdd2901325e5693db1f67215a6beb741f7fedd36c39de WatchSource:0}: Error finding container 7d7539daaef63570cb9fdd2901325e5693db1f67215a6beb741f7fedd36c39de: Status 404 returned error can't find the container with id 7d7539daaef63570cb9fdd2901325e5693db1f67215a6beb741f7fedd36c39de
Nov 02 01:37:23.904554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.904534    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"]
Nov 02 01:37:23.907254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.907223    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:23.907386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.907349    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:23.907432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:23.907388    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:24.110719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:24.110682    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:24.283534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:24.283493    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" event={"ID":"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead","Type":"ContainerStarted","Data":"d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df"}
Nov 02 01:37:24.283728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:24.283597    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" event={"ID":"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead","Type":"ContainerStarted","Data":"7d7539daaef63570cb9fdd2901325e5693db1f67215a6beb741f7fedd36c39de"}
Nov 02 01:37:24.310152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:24.310102    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podStartSLOduration=2.31008775 podStartE2EDuration="2.31008775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:24.306689365 +0000 UTC m=+1349.611622192" watchObservedRunningTime="2026-11-02 01:37:24.31008775 +0000 UTC m=+1349.615020630"
Nov 02 01:37:24.457406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:24.452683    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:24.469322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:24.459417    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 01:37:24.469322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:24.459481    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 01:37:24.885649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:24.885614    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:25.454251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:25.454219    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:25.944982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:25.944947    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:26.301916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:26.301825    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:26.302846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:26.302824    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:37:26.882579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:26.882544    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:27.147367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:27.128849    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:27.877906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:27.877876    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:28.186059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.186011    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:28.244905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.244866    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"]
Nov 02 01:37:28.250276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.250249    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:37:28.262712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.262686    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f689bc-dockercfg-dmsvj\""
Nov 02 01:37:28.264046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.264023    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f689bc\""
Nov 02 01:37:28.266404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.266383    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"]
Nov 02 01:37:28.271192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.271171    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"]
Nov 02 01:37:28.276351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.276329    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:37:28.292838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.292776    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"]
Nov 02 01:37:28.341567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.341504    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7739d3-8fb8-453e-8af5-345d8bd842e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z\" (UID: \"0f7739d3-8fb8-453e-8af5-345d8bd842e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:37:28.341769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.341573    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2nd7\" (UniqueName: \"kubernetes.io/projected/90d90625-4204-4fec-8fb1-1237f358bd6a-kube-api-access-m2nd7\") pod \"payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx\" (UID: \"90d90625-4204-4fec-8fb1-1237f358bd6a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:37:28.341769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.341725    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d90625-4204-4fec-8fb1-1237f358bd6a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx\" (UID: \"90d90625-4204-4fec-8fb1-1237f358bd6a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:37:28.341903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.341863    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nlk2\" (UniqueName: \"kubernetes.io/projected/0f7739d3-8fb8-453e-8af5-345d8bd842e4-kube-api-access-7nlk2\") pod \"payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z\" (UID: \"0f7739d3-8fb8-453e-8af5-345d8bd842e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:37:28.455371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.451303    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nlk2\" (UniqueName: \"kubernetes.io/projected/0f7739d3-8fb8-453e-8af5-345d8bd842e4-kube-api-access-7nlk2\") pod \"payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z\" (UID: \"0f7739d3-8fb8-453e-8af5-345d8bd842e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:37:28.455371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.451536    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7739d3-8fb8-453e-8af5-345d8bd842e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z\" (UID: \"0f7739d3-8fb8-453e-8af5-345d8bd842e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:37:28.455371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.451601    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2nd7\" (UniqueName: \"kubernetes.io/projected/90d90625-4204-4fec-8fb1-1237f358bd6a-kube-api-access-m2nd7\") pod \"payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx\" (UID: \"90d90625-4204-4fec-8fb1-1237f358bd6a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:37:28.455371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.451741    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d90625-4204-4fec-8fb1-1237f358bd6a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx\" (UID: \"90d90625-4204-4fec-8fb1-1237f358bd6a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:37:28.545210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.532111    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d90625-4204-4fec-8fb1-1237f358bd6a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx\" (UID: \"90d90625-4204-4fec-8fb1-1237f358bd6a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:37:28.545210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.534946    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7739d3-8fb8-453e-8af5-345d8bd842e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z\" (UID: \"0f7739d3-8fb8-453e-8af5-345d8bd842e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:37:28.579457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.575001    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2nd7\" (UniqueName: \"kubernetes.io/projected/90d90625-4204-4fec-8fb1-1237f358bd6a-kube-api-access-m2nd7\") pod \"payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx\" (UID: \"90d90625-4204-4fec-8fb1-1237f358bd6a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:37:28.579457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.575514    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:37:28.612327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.593761    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:28.621928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.619965    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nlk2\" (UniqueName: \"kubernetes.io/projected/0f7739d3-8fb8-453e-8af5-345d8bd842e4-kube-api-access-7nlk2\") pod \"payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z\" (UID: \"0f7739d3-8fb8-453e-8af5-345d8bd842e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:37:28.910436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:28.909021    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:37:29.048315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:29.040993    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:29.584265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:29.584215    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"]
Nov 02 01:37:29.598426 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:37:29.595949    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d90625_4204_4fec_8fb1_1237f358bd6a.slice/crio-b2cddfa875dce68456b47a75693e8a8788cd3808628e5db187e973e423e238af WatchSource:0}: Error finding container b2cddfa875dce68456b47a75693e8a8788cd3808628e5db187e973e423e238af: Status 404 returned error can't find the container with id b2cddfa875dce68456b47a75693e8a8788cd3808628e5db187e973e423e238af
Nov 02 01:37:29.680809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:29.680783    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"]
Nov 02 01:37:29.682674 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:37:29.682648    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7739d3_8fb8_453e_8af5_345d8bd842e4.slice/crio-91631e2e26e2f1ebdc1a7dc9a60d5854615b6015baf139657e938e009affecc2 WatchSource:0}: Error finding container 91631e2e26e2f1ebdc1a7dc9a60d5854615b6015baf139657e938e009affecc2: Status 404 returned error can't find the container with id 91631e2e26e2f1ebdc1a7dc9a60d5854615b6015baf139657e938e009affecc2
Nov 02 01:37:30.037406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:30.036914    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:30.792328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:30.786913    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:37:30.819752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:30.819710    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx" event={"ID":"90d90625-4204-4fec-8fb1-1237f358bd6a","Type":"ContainerStarted","Data":"a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec"}
Nov 02 01:37:30.820053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:30.820034    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx" event={"ID":"90d90625-4204-4fec-8fb1-1237f358bd6a","Type":"ContainerStarted","Data":"b2cddfa875dce68456b47a75693e8a8788cd3808628e5db187e973e423e238af"}
Nov 02 01:37:30.828484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:30.828456    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:37:30.838065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:30.838029    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z" event={"ID":"0f7739d3-8fb8-453e-8af5-345d8bd842e4","Type":"ContainerStarted","Data":"42a03eccb342b409f039e155501c3bdcbb7662883e12e09bd6a1d3281fda6773"}
Nov 02 01:37:30.838330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:30.838307    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z" event={"ID":"0f7739d3-8fb8-453e-8af5-345d8bd842e4","Type":"ContainerStarted","Data":"91631e2e26e2f1ebdc1a7dc9a60d5854615b6015baf139657e938e009affecc2"}
Nov 02 01:37:30.872540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:30.872517    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:37:30.908828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:30.908762    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx" podStartSLOduration=2.908740232 podStartE2EDuration="2.908740232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:30.888571881 +0000 UTC m=+1356.193504707" watchObservedRunningTime="2026-11-02 01:37:30.908740232 +0000 UTC m=+1356.213673058"
Nov 02 01:37:30.916957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:30.916918    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:31.044486 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:31.044393    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:31.045312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:31.045254    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:31.172511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:31.172461    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:31.347547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:31.347448    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:31.435980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:31.435945    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:31.539314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:31.538944    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:31.628370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:31.624729    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:37:31.714321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:31.707304    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:37:32.085280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:32.085246    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:32.418134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:32.418104    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:33.448316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:33.447819    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:34.281978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:34.281942    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:34.988223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:34.988188    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:35.280373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:35.278568    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:36.181770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:36.181736    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:36.982055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:36.982022    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:37.765702 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:37.765659    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:37.984317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:37.980556    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:38.980349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:38.980211    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:39.783366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:39.783323    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:40.707927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:40.707879    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:41.459603 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:41.459570    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:41.867898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:41.867819    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:42.325883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:42.325848    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:42.557735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:42.557688    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:42.792422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:42.790576    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:43.056954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:43.056853    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:37:43.058120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:43.058100    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:37:43.137316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:43.120129    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z" podStartSLOduration=15.120104977 podStartE2EDuration="15.120104977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:30.933779607 +0000 UTC m=+1356.238712434" watchObservedRunningTime="2026-11-02 01:37:43.120104977 +0000 UTC m=+1368.425037803"
Nov 02 01:37:43.388052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:43.387950    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:43.692536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:43.692500    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:44.410535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:44.410503    4851 generic.go:356] "Generic (PLEG): container finished" podID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerID="eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80" exitCode=0
Nov 02 01:37:44.411014 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:44.410577    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" event={"ID":"6bda2b76-d865-47d7-9c84-e01ffbd776f4","Type":"ContainerDied","Data":"eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80"}
Nov 02 01:37:44.411088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:44.411064    4851 scope.go:122] "RemoveContainer" containerID="eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80"
Nov 02 01:37:44.413000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:44.412971    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:44.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:44.413054    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:44.413109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:44.413098    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:44.723533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:44.723498    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:45.421548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:45.421512    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" event={"ID":"6bda2b76-d865-47d7-9c84-e01ffbd776f4","Type":"ContainerStarted","Data":"8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121"}
Nov 02 01:37:45.642314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:45.636388    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:37:45.910997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:45.910964    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:37:45.912615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:45.912585    4851 patch_prober.go:29] interesting pod/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 01:37:45.912757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:45.912641    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 01:37:46.096824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:46.096789    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:46.324722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:46.324631    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:46.519578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:46.519541    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:46.911154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:46.911100    4851 patch_prober.go:29] interesting pod/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 01:37:46.911476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:46.911154    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 01:37:47.003307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:47.003255    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:47.357378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:47.356713    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:47.586239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:47.586207    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:47.620626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:47.620545    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"]
Nov 02 01:37:47.625988 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:47.625922    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:37:47.658313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:47.633345    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-12e7cb\""
Nov 02 01:37:47.674855 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:47.660595    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-12e7cb-dockercfg-ztqr7\""
Nov 02 01:37:47.695027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:47.686795    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"]
Nov 02 01:37:47.892660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:47.814276    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pbw\" (UniqueName: \"kubernetes.io/projected/fe26ff80-ecf0-4541-adc0-56877c9c009d-kube-api-access-j5pbw\") pod \"payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5\" (UID: \"fe26ff80-ecf0-4541-adc0-56877c9c009d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:37:48.020842 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.020789    4851 patch_prober.go:29] interesting pod/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 01:37:48.039946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.021453    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe26ff80-ecf0-4541-adc0-56877c9c009d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5\" (UID: \"fe26ff80-ecf0-4541-adc0-56877c9c009d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:37:48.047418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.040542    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 01:37:48.047894 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:37:48.022609    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:37:48.112607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.112571    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:48.149375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.148273    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"]
Nov 02 01:37:48.176319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.171969    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe26ff80-ecf0-4541-adc0-56877c9c009d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5\" (UID: \"fe26ff80-ecf0-4541-adc0-56877c9c009d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:37:48.176319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.172272    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pbw\" (UniqueName: \"kubernetes.io/projected/fe26ff80-ecf0-4541-adc0-56877c9c009d-kube-api-access-j5pbw\") pod \"payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5\" (UID: \"fe26ff80-ecf0-4541-adc0-56877c9c009d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:37:48.176319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.173701    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe26ff80-ecf0-4541-adc0-56877c9c009d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5\" (UID: \"fe26ff80-ecf0-4541-adc0-56877c9c009d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:37:48.229975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.229948    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:37:48.232363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.232331    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pbw\" (UniqueName: \"kubernetes.io/projected/fe26ff80-ecf0-4541-adc0-56877c9c009d-kube-api-access-j5pbw\") pod \"payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5\" (UID: \"fe26ff80-ecf0-4541-adc0-56877c9c009d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:37:48.259326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.257080    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"]
Nov 02 01:37:48.316379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.315819    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:37:48.371065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.371029    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:48.462190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.461435    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3daa532-ae71-4a64-a684-59d4fbb22088-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf\" (UID: \"b3daa532-ae71-4a64-a684-59d4fbb22088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:37:48.462190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.461944    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgvzr\" (UniqueName: \"kubernetes.io/projected/b3daa532-ae71-4a64-a684-59d4fbb22088-kube-api-access-pgvzr\") pod \"payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf\" (UID: \"b3daa532-ae71-4a64-a684-59d4fbb22088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:37:48.592316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.591824    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvzr\" (UniqueName: \"kubernetes.io/projected/b3daa532-ae71-4a64-a684-59d4fbb22088-kube-api-access-pgvzr\") pod \"payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf\" (UID: \"b3daa532-ae71-4a64-a684-59d4fbb22088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:37:48.592316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.592013    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3daa532-ae71-4a64-a684-59d4fbb22088-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf\" (UID: \"b3daa532-ae71-4a64-a684-59d4fbb22088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:37:48.605221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.593159    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3daa532-ae71-4a64-a684-59d4fbb22088-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf\" (UID: \"b3daa532-ae71-4a64-a684-59d4fbb22088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:37:48.636337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.627970    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvzr\" (UniqueName: \"kubernetes.io/projected/b3daa532-ae71-4a64-a684-59d4fbb22088-kube-api-access-pgvzr\") pod \"payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf\" (UID: \"b3daa532-ae71-4a64-a684-59d4fbb22088\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:37:48.636337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.629605    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:37:48.719210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.716491    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:48.903781 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:37:48.903736    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe26ff80_ecf0_4541_adc0_56877c9c009d.slice/crio-6abcdeeab0a679c3424b47c81c1efa0b03183c58cf528e7b2b3d441abb9ab8bc WatchSource:0}: Error finding container 6abcdeeab0a679c3424b47c81c1efa0b03183c58cf528e7b2b3d441abb9ab8bc: Status 404 returned error can't find the container with id 6abcdeeab0a679c3424b47c81c1efa0b03183c58cf528e7b2b3d441abb9ab8bc
Nov 02 01:37:48.905097 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.905064    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"]
Nov 02 01:37:48.912955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.912825    4851 patch_prober.go:29] interesting pod/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 01:37:48.912955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:48.912882    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 01:37:49.470658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.470620    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"]
Nov 02 01:37:49.473723 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:37:49.473692    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3daa532_ae71_4a64_a684_59d4fbb22088.slice/crio-ffd567846d7a1a026e658ebc593be44567ad4540b33e764458e5a5c6e9fd42e7 WatchSource:0}: Error finding container ffd567846d7a1a026e658ebc593be44567ad4540b33e764458e5a5c6e9fd42e7: Status 404 returned error can't find the container with id ffd567846d7a1a026e658ebc593be44567ad4540b33e764458e5a5c6e9fd42e7
Nov 02 01:37:49.478986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.478958    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:49.800327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.800203    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5" event={"ID":"fe26ff80-ecf0-4541-adc0-56877c9c009d","Type":"ContainerStarted","Data":"cd55ebf79dbc04e80f5fa0dbff509bc063845c65759293d3ffa8a08e0c65ee71"}
Nov 02 01:37:49.800916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.800889    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5" event={"ID":"fe26ff80-ecf0-4541-adc0-56877c9c009d","Type":"ContainerStarted","Data":"6abcdeeab0a679c3424b47c81c1efa0b03183c58cf528e7b2b3d441abb9ab8bc"}
Nov 02 01:37:49.802690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.802371    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:37:49.805668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.805643    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf" event={"ID":"b3daa532-ae71-4a64-a684-59d4fbb22088","Type":"ContainerStarted","Data":"6a74d7dcc7cd4f0112f3400d171f6b96d0fa7b6fb9031a72fc9b9e160639b5f5"}
Nov 02 01:37:49.805871 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.805850    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf" event={"ID":"b3daa532-ae71-4a64-a684-59d4fbb22088","Type":"ContainerStarted","Data":"ffd567846d7a1a026e658ebc593be44567ad4540b33e764458e5a5c6e9fd42e7"}
Nov 02 01:37:49.806943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.806920    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:37:49.829808 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.829704    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5" podStartSLOduration=2.829688705 podStartE2EDuration="2.829688705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:49.821676279 +0000 UTC m=+1375.126609119" watchObservedRunningTime="2026-11-02 01:37:49.829688705 +0000 UTC m=+1375.134621530"
Nov 02 01:37:49.864431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.864371    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf" podStartSLOduration=2.864356501 podStartE2EDuration="2.864356501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:49.858764764 +0000 UTC m=+1375.163697583" watchObservedRunningTime="2026-11-02 01:37:49.864356501 +0000 UTC m=+1375.169289327"
Nov 02 01:37:49.911602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.911509    4851 patch_prober.go:29] interesting pod/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 01:37:49.911602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:49.911582    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 01:37:50.487819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:50.487781    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:50.911077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:50.911047    4851 patch_prober.go:29] interesting pod/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 01:37:50.911572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:50.911097    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 01:37:51.024555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.024517    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"]
Nov 02 01:37:51.029136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.029107    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75fc18b-v8t88"]
Nov 02 01:37:51.186001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.185918    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:51.579944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.579864    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"]
Nov 02 01:37:51.584905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.584888    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.595561 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.595536    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ffe5c1-openshift-default-dockercfg-f9w6k\""
Nov 02 01:37:51.600336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.600276    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"]
Nov 02 01:37:51.743033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.742993    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-envoy\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.743209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.743118    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-token\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.743209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.743147    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-credential-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.743401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.743285    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istiod-ca-cert\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.743471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.743438    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-data\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.743563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.743547    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-podinfo\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.743650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.743629    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-certs\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.743723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.743708    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x669\" (UniqueName: \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-kube-api-access-2x669\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.743836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.743815    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.846991    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-podinfo\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847070    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-certs\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847109    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x669\" (UniqueName: \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-kube-api-access-2x669\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847184    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847339    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-envoy\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847397    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-token\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847426    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-credential-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847492    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istiod-ca-cert\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847567    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-data\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847571    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-certs\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847628    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847875    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-credential-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.847983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.847964    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-data\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.848198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.848180    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istiod-ca-cert\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.850094 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.850061    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-envoy\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.850512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.850493    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-podinfo\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.859473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.859448    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-token\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.859591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.859578    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x669\" (UniqueName: \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-kube-api-access-2x669\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.903961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.903921    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:51.921757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.921724    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:37:51.923397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.923365    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:37:51.927114 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:51.927090    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:52.185177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.183573    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:52.476656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.476612    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:52.606850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.604576    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:52.606850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.604701    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:52.606850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.604756    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:37:52.642599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.642531    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"]
Nov 02 01:37:52.866205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.866164    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ea69e3-38be-469c-b453-eab809497a72" path="/var/lib/kubelet/pods/d2ea69e3-38be-469c-b453-eab809497a72/volumes"
Nov 02 01:37:52.874890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.874759    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" event={"ID":"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2","Type":"ContainerStarted","Data":"9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850"}
Nov 02 01:37:52.874890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.874841    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" event={"ID":"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2","Type":"ContainerStarted","Data":"5cb4aca257aae58f04f3635fd85a8ac58ef952453621e1d8cc3023ecb693a0a6"}
Nov 02 01:37:52.901913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.901849    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" podStartSLOduration=1.901830414 podStartE2EDuration="1.901830414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:52.897516563 +0000 UTC m=+1378.202449429" watchObservedRunningTime="2026-11-02 01:37:52.901830414 +0000 UTC m=+1378.206763241"
Nov 02 01:37:52.904239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:52.904213    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:53.031011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:53.030913    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:53.394982 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:53.394790    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:53.905612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:53.905576    4851 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 01:37:53.905802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:53.905628    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 01:37:53.949854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:53.949819    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:54.904189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:54.904158    4851 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 01:37:54.904627 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:54.904225    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 01:37:54.965988 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:54.965928    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:55.872718 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:55.872683    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:55.904554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:55.904522    4851 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 01:37:55.904909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:55.904582    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 01:37:56.783986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:56.783956    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:56.919327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:56.909003    4851 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 01:37:56.930911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:56.930857    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 01:37:57.602005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:57.601967    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:57.904868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:57.904782    4851 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 01:37:57.904868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:57.904848    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 01:37:58.455315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:58.454182    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:58.674966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:58.674931    4851 scope.go:122] "RemoveContainer" containerID="0db41ee6e0eadfdd369f523e94bc5fec0e4f36b22b5ec693f518b83fe29b8f4f"
Nov 02 01:37:58.688014 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:58.687994    4851 scope.go:122] "RemoveContainer" containerID="abd4720485ceb602748c6203942d4c1396a3dcf5120757c48c8826d41b6642e1"
Nov 02 01:37:58.699458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:58.699438    4851 scope.go:122] "RemoveContainer" containerID="35e084fbb0cb99f30bf3b88fbc293be2fcf4ff8b67f273dc84c2053ccfbe8f7f"
Nov 02 01:37:58.713271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:58.713247    4851 scope.go:122] "RemoveContainer" containerID="58bffe8bdcaf308b8a46d741137e9727eb4a93c39a944febb23ac270cfbd1a1e"
Nov 02 01:37:58.904384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:58.904346    4851 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 01:37:58.904595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:58.904409    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 01:37:59.281481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:59.281443    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:59.610428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:59.610347    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:59.815422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:59.815390    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:37:59.911558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:59.911533    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:59.912669 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:59.912647    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:59.913852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:59.913831    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:37:59.960120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:37:59.960085    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"]
Nov 02 01:38:00.042090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:00.042055    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:00.103750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:00.103711    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy" containerID="cri-o://ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12" gracePeriod=30
Nov 02 01:38:00.326937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:00.326846    4851 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log" limit=1024
Nov 02 01:38:01.461592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.461555    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"]
Nov 02 01:38:01.468904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.468408    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.472619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.472588    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:38:01.473670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.473637    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:38:01.473824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.473678    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:38:01.473824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.473699    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:38:01.474921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.474885    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"]
Nov 02 01:38:01.580649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.580609    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16a36ae9-988pj\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.580649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.580655    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd52p\" (UniqueName: \"kubernetes.io/projected/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-kube-api-access-fd52p\") pod \"maas-api-revoke-sub-16a36ae9-988pj\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.580994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.580912    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16a36ae9-988pj\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.683234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.683191    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16a36ae9-988pj\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.683234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.683240    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd52p\" (UniqueName: \"kubernetes.io/projected/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-kube-api-access-fd52p\") pod \"maas-api-revoke-sub-16a36ae9-988pj\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.683469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.683438    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16a36ae9-988pj\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.684092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.684022    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16a36ae9-988pj\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.685813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.685791    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16a36ae9-988pj\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.695731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.695699    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd52p\" (UniqueName: \"kubernetes.io/projected/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-kube-api-access-fd52p\") pod \"maas-api-revoke-sub-16a36ae9-988pj\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.786653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.786563    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:01.856356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.856260    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:38:01.857280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:01.857258    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:38:02.078088 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:38:02.078051    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe843ac8_a6f4_422c_9fd9_57d4e18ef1b6.slice/crio-2207b572ab7a2486d306884a452f7cc2f73a170c99f0950d072179c6944e4e8f WatchSource:0}: Error finding container 2207b572ab7a2486d306884a452f7cc2f73a170c99f0950d072179c6944e4e8f: Status 404 returned error can't find the container with id 2207b572ab7a2486d306884a452f7cc2f73a170c99f0950d072179c6944e4e8f
Nov 02 01:38:02.078818 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:02.078788    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"]
Nov 02 01:38:02.120431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:02.120399    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj" event={"ID":"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6","Type":"ContainerStarted","Data":"2207b572ab7a2486d306884a452f7cc2f73a170c99f0950d072179c6944e4e8f"}
Nov 02 01:38:02.530750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:02.530699    4851 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:38:02.530750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:02.530775    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:38:03.181147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:03.181115    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj" event={"ID":"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6","Type":"ContainerStarted","Data":"9da8e1663b38c77998abbb6a53c11a842ef0de9236786cb8cbbf97a363749fc9"}
Nov 02 01:38:04.191269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.191237    4851 generic.go:356] "Generic (PLEG): container finished" podID="be843ac8-a6f4-422c-9fd9-57d4e18ef1b6" containerID="9da8e1663b38c77998abbb6a53c11a842ef0de9236786cb8cbbf97a363749fc9" exitCode=0
Nov 02 01:38:04.191721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.191332    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj" event={"ID":"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6","Type":"ContainerDied","Data":"9da8e1663b38c77998abbb6a53c11a842ef0de9236786cb8cbbf97a363749fc9"}
Nov 02 01:38:04.373473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.373442    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:04.463924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.463823    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-openshift-ca\") pod \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") "
Nov 02 01:38:04.464108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.464091    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-xks-ca\") pod \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") "
Nov 02 01:38:04.464219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.464189    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-openshift-ca" pod "be843ac8-a6f4-422c-9fd9-57d4e18ef1b6" (UID: "be843ac8-a6f4-422c-9fd9-57d4e18ef1b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:38:04.464219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.464192    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-kube-api-access-fd52p\" (UniqueName: \"kubernetes.io/projected/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-kube-api-access-fd52p\") pod \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\" (UID: \"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6\") "
Nov 02 01:38:04.465525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.465498    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:04.466618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.466591    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-xks-ca" pod "be843ac8-a6f4-422c-9fd9-57d4e18ef1b6" (UID: "be843ac8-a6f4-422c-9fd9-57d4e18ef1b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:38:04.466719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.466640    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-kube-api-access-fd52p" pod "be843ac8-a6f4-422c-9fd9-57d4e18ef1b6" (UID: "be843ac8-a6f4-422c-9fd9-57d4e18ef1b6"). InnerVolumeSpecName "kube-api-access-fd52p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:38:04.567562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.567530    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:04.567562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:04.567559    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd52p\" (UniqueName: \"kubernetes.io/projected/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6-kube-api-access-fd52p\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:05.026800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.026715    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"]
Nov 02 01:38:05.031351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.031326    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c918649-6jq6p"]
Nov 02 01:38:05.201402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.201371    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"
Nov 02 01:38:05.201793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.201405    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj" event={"ID":"be843ac8-a6f4-422c-9fd9-57d4e18ef1b6","Type":"ContainerDied","Data":"2207b572ab7a2486d306884a452f7cc2f73a170c99f0950d072179c6944e4e8f"}
Nov 02 01:38:05.201793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.201450    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2207b572ab7a2486d306884a452f7cc2f73a170c99f0950d072179c6944e4e8f"
Nov 02 01:38:05.427985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.427957    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:38:05.499030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.498984    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-socket\") pod \"fb40876a-6ba0-40ee-864a-103d087b918e\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") "
Nov 02 01:38:05.499381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.499360    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-envoy\") pod \"fb40876a-6ba0-40ee-864a-103d087b918e\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") "
Nov 02 01:38:05.499640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.499624    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-credential-socket\") pod \"fb40876a-6ba0-40ee-864a-103d087b918e\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") "
Nov 02 01:38:05.499766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.499753    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb40876a-6ba0-40ee-864a-103d087b918e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb40876a-6ba0-40ee-864a-103d087b918e-istiod-ca-cert\") pod \"fb40876a-6ba0-40ee-864a-103d087b918e\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") "
Nov 02 01:38:05.499886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.499871    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-data\") pod \"fb40876a-6ba0-40ee-864a-103d087b918e\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") "
Nov 02 01:38:05.500210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.500187    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-kube-api-access-b2t4j\" (UniqueName: \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-kube-api-access-b2t4j\") pod \"fb40876a-6ba0-40ee-864a-103d087b918e\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") "
Nov 02 01:38:05.500602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.500576    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-istio-token\" (UniqueName: \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-istio-token\") pod \"fb40876a-6ba0-40ee-864a-103d087b918e\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") "
Nov 02 01:38:05.500836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.500818    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb40876a-6ba0-40ee-864a-103d087b918e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb40876a-6ba0-40ee-864a-103d087b918e-istio-podinfo\") pod \"fb40876a-6ba0-40ee-864a-103d087b918e\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") "
Nov 02 01:38:05.500987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.500972    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-certs\") pod \"fb40876a-6ba0-40ee-864a-103d087b918e\" (UID: \"fb40876a-6ba0-40ee-864a-103d087b918e\") "
Nov 02 01:38:05.502628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.502604    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-certs" pod "fb40876a-6ba0-40ee-864a-103d087b918e" (UID: "fb40876a-6ba0-40ee-864a-103d087b918e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:38:05.503018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.502998    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-socket" pod "fb40876a-6ba0-40ee-864a-103d087b918e" (UID: "fb40876a-6ba0-40ee-864a-103d087b918e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:38:05.527377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.516471    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-credential-socket" pod "fb40876a-6ba0-40ee-864a-103d087b918e" (UID: "fb40876a-6ba0-40ee-864a-103d087b918e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:38:05.527377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.516804    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-data" pod "fb40876a-6ba0-40ee-864a-103d087b918e" (UID: "fb40876a-6ba0-40ee-864a-103d087b918e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:38:05.527377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.517162    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb40876a-6ba0-40ee-864a-103d087b918e-istiod-ca-cert" pod "fb40876a-6ba0-40ee-864a-103d087b918e" (UID: "fb40876a-6ba0-40ee-864a-103d087b918e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:38:05.618251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.615392    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:05.618251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.615423    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:05.618251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.615439    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb40876a-6ba0-40ee-864a-103d087b918e-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:05.618251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.615453    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:05.618251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.615466    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:05.626033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.625953    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-kube-api-access-b2t4j" pod "fb40876a-6ba0-40ee-864a-103d087b918e" (UID: "fb40876a-6ba0-40ee-864a-103d087b918e"). InnerVolumeSpecName "kube-api-access-b2t4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:38:05.626033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.625976    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-envoy" pod "fb40876a-6ba0-40ee-864a-103d087b918e" (UID: "fb40876a-6ba0-40ee-864a-103d087b918e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:38:05.626254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.626031    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-istio-token" pod "fb40876a-6ba0-40ee-864a-103d087b918e" (UID: "fb40876a-6ba0-40ee-864a-103d087b918e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:38:05.626803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.626761    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb40876a-6ba0-40ee-864a-103d087b918e-istio-podinfo" pod "fb40876a-6ba0-40ee-864a-103d087b918e" (UID: "fb40876a-6ba0-40ee-864a-103d087b918e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:38:05.717582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.717544    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb40876a-6ba0-40ee-864a-103d087b918e-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:05.717582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.717577    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb40876a-6ba0-40ee-864a-103d087b918e-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:05.717582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.717587    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2t4j\" (UniqueName: \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-kube-api-access-b2t4j\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:05.717820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:05.717597    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb40876a-6ba0-40ee-864a-103d087b918e-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:38:06.212069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.212035    4851 generic.go:356] "Generic (PLEG): container finished" podID="fb40876a-6ba0-40ee-864a-103d087b918e" containerID="ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12" exitCode=0
Nov 02 01:38:06.212069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.212073    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" event={"ID":"fb40876a-6ba0-40ee-864a-103d087b918e","Type":"ContainerDied","Data":"ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12"}
Nov 02 01:38:06.212635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.212112    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"
Nov 02 01:38:06.212635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.212123    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948" event={"ID":"fb40876a-6ba0-40ee-864a-103d087b918e","Type":"ContainerDied","Data":"cf4245ba5461575f8f358469e0a61e4105efb708fe34dc85a4cd169c9f2eb3ee"}
Nov 02 01:38:06.212635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.212146    4851 scope.go:122] "RemoveContainer" containerID="ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12"
Nov 02 01:38:06.225384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.225133    4851 scope.go:122] "RemoveContainer" containerID="54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e"
Nov 02 01:38:06.237356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.237334    4851 scope.go:122] "RemoveContainer" containerID="ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12"
Nov 02 01:38:06.237691 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:38:06.237670    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12\": container with ID starting with ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12 not found: ID does not exist" containerID="ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12"
Nov 02 01:38:06.237742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.237710    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12"} err="failed to get container status \"ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12\": rpc error: code = NotFound desc = could not find container \"ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12\": container with ID starting with ad604b6404f565d65eff1be14a88eb99d331286fa8b96108a4eb5ad24ea5ab12 not found: ID does not exist"
Nov 02 01:38:06.237742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.237728    4851 scope.go:122] "RemoveContainer" containerID="54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e"
Nov 02 01:38:06.237986 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:38:06.237966    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e\": container with ID starting with 54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e not found: ID does not exist" containerID="54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e"
Nov 02 01:38:06.238060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.237995    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e"} err="failed to get container status \"54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e\": rpc error: code = NotFound desc = could not find container \"54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e\": container with ID starting with 54eae01171a87e592f1aaf875648c41431b908721fe2e416cb326ac5eb357e7e not found: ID does not exist"
Nov 02 01:38:06.240964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.240931    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"]
Nov 02 01:38:06.245387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.245364    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-p6948"]
Nov 02 01:38:06.849868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.849838    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465ac986-4ebc-4351-b221-51e64500cadb" path="/var/lib/kubelet/pods/465ac986-4ebc-4351-b221-51e64500cadb/volumes"
Nov 02 01:38:06.850517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:06.850496    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" path="/var/lib/kubelet/pods/fb40876a-6ba0-40ee-864a-103d087b918e/volumes"
Nov 02 01:38:11.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.773662    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"]
Nov 02 01:38:11.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.775055    4851 cpu_manager.go:414] "containerMap: removing container" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy"
Nov 02 01:38:11.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.775078    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy"
Nov 02 01:38:11.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.775097    4851 cpu_manager.go:414] "containerMap: removing container" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy"
Nov 02 01:38:11.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.775104    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy"
Nov 02 01:38:11.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.775135    4851 cpu_manager.go:414] "containerMap: removing container" podUID="be843ac8-a6f4-422c-9fd9-57d4e18ef1b6" containerName="revoke-keys"
Nov 02 01:38:11.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.775142    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be843ac8-a6f4-422c-9fd9-57d4e18ef1b6" containerName="revoke-keys"
Nov 02 01:38:11.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.775357    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy"
Nov 02 01:38:11.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.775376    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb40876a-6ba0-40ee-864a-103d087b918e" containerName="istio-proxy"
Nov 02 01:38:11.784318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.775389    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="be843ac8-a6f4-422c-9fd9-57d4e18ef1b6" containerName="revoke-keys"
Nov 02 01:38:11.799282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.799251    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:11.834312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.830508    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-edef28-openshift-default-dockercfg-bg4zb\""
Nov 02 01:38:11.869651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.840952    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"]
Nov 02 01:38:12.164325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:11.935354    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.176027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.174874    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.176027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.174999    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.176027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.175171    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.176027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.175217    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548143a8-0c95-41f1-9c60-857747c3c4dd-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.176027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.175270    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.176027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.175369    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.182167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.176662    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.182421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.182366    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqklt\" (UniqueName: \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-kube-api-access-vqklt\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.289634 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.289595    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqklt\" (UniqueName: \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-kube-api-access-vqklt\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.293124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.293085    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.293453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.293433    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.293735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.293719    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.294126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.294101    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.294367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.294348    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548143a8-0c95-41f1-9c60-857747c3c4dd-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.294508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.294492    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.294728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.294710    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.314378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.295124    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.315111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.311611    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.321940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.313036    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.322243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.313714    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548143a8-0c95-41f1-9c60-857747c3c4dd-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.322390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.315973    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqklt\" (UniqueName: \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-kube-api-access-vqklt\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.322527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.317006    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.322730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.318344    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.322865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.319364    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.325927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.319890    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.326908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.326883    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.486398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.486339    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:12.790871 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:38:12.790828    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548143a8_0c95_41f1_9c60_857747c3c4dd.slice/crio-910adbc47490f2b921cbd316dfce5482881460b445363522a0cdcb5c31736ea1 WatchSource:0}: Error finding container 910adbc47490f2b921cbd316dfce5482881460b445363522a0cdcb5c31736ea1: Status 404 returned error can't find the container with id 910adbc47490f2b921cbd316dfce5482881460b445363522a0cdcb5c31736ea1
Nov 02 01:38:12.792786 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.792757    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"]
Nov 02 01:38:12.801033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.799545    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:38:12.801033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.799642    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:38:12.801033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:12.799684    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:38:13.368868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:13.368822    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" event={"ID":"548143a8-0c95-41f1-9c60-857747c3c4dd","Type":"ContainerStarted","Data":"f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85"}
Nov 02 01:38:13.369116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:13.368889    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" event={"ID":"548143a8-0c95-41f1-9c60-857747c3c4dd","Type":"ContainerStarted","Data":"910adbc47490f2b921cbd316dfce5482881460b445363522a0cdcb5c31736ea1"}
Nov 02 01:38:13.397942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:13.397887    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podStartSLOduration=2.397872063 podStartE2EDuration="2.397872063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:13.39266768 +0000 UTC m=+1398.697600541" watchObservedRunningTime="2026-11-02 01:38:13.397872063 +0000 UTC m=+1398.702804889"
Nov 02 01:38:13.486969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:13.486934    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:14.493823 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:14.493795    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:15.396664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:15.396631    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:15.397737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:15.397709    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:38:17.336498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.336463    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"]
Nov 02 01:38:17.341763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.341744    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:38:17.349146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.349122    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"]
Nov 02 01:38:17.350245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.350212    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-edef28\""
Nov 02 01:38:17.434708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.434675    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clh9p\" (UniqueName: \"kubernetes.io/projected/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-kube-api-access-clh9p\") pod \"payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8\" (UID: \"d9c26f79-3936-4990-8b0a-3b7e2b32de0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:38:17.434921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.434902    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8\" (UID: \"d9c26f79-3936-4990-8b0a-3b7e2b32de0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:38:17.537138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.537104    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clh9p\" (UniqueName: \"kubernetes.io/projected/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-kube-api-access-clh9p\") pod \"payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8\" (UID: \"d9c26f79-3936-4990-8b0a-3b7e2b32de0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:38:17.537619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.537585    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8\" (UID: \"d9c26f79-3936-4990-8b0a-3b7e2b32de0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:38:17.538596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.538573    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8\" (UID: \"d9c26f79-3936-4990-8b0a-3b7e2b32de0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:38:17.550357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.550331    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clh9p\" (UniqueName: \"kubernetes.io/projected/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-kube-api-access-clh9p\") pod \"payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8\" (UID: \"d9c26f79-3936-4990-8b0a-3b7e2b32de0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:38:17.666371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:17.663591    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:38:18.102360 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:38:18.096512    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:38:18.372338 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:38:18.366794    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c26f79_3936_4990_8b0a_3b7e2b32de0e.slice/crio-0174dc1549c1d1453a249e028baa2b5784debdca8a800f3f6c57747e9311d6b0 WatchSource:0}: Error finding container 0174dc1549c1d1453a249e028baa2b5784debdca8a800f3f6c57747e9311d6b0: Status 404 returned error can't find the container with id 0174dc1549c1d1453a249e028baa2b5784debdca8a800f3f6c57747e9311d6b0
Nov 02 01:38:18.372338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:18.367476    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"]
Nov 02 01:38:18.428877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:18.428840    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8" event={"ID":"d9c26f79-3936-4990-8b0a-3b7e2b32de0e","Type":"ContainerStarted","Data":"0174dc1549c1d1453a249e028baa2b5784debdca8a800f3f6c57747e9311d6b0"}
Nov 02 01:38:19.038740 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.038705    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"]
Nov 02 01:38:19.044918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.044892    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:38:19.045393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.045369    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"]
Nov 02 01:38:19.049950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.049931    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:38:19.053784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.053725    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8b3bae\""
Nov 02 01:38:19.053923 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.053887    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8b3bae-dockercfg-psw6m\""
Nov 02 01:38:19.060191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.060165    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"]
Nov 02 01:38:19.102868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.101154    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"]
Nov 02 01:38:19.377376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.243932    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be589a6-1443-4701-a4e6-0a5789a69eec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt\" (UID: \"5be589a6-1443-4701-a4e6-0a5789a69eec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:38:19.377376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.244080    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlsl6\" (UniqueName: \"kubernetes.io/projected/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-kube-api-access-jlsl6\") pod \"payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k\" (UID: \"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:38:19.377376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.244119    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4scpq\" (UniqueName: \"kubernetes.io/projected/5be589a6-1443-4701-a4e6-0a5789a69eec-kube-api-access-4scpq\") pod \"payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt\" (UID: \"5be589a6-1443-4701-a4e6-0a5789a69eec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:38:19.377376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.244190    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k\" (UID: \"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:38:19.377376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.360768    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be589a6-1443-4701-a4e6-0a5789a69eec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt\" (UID: \"5be589a6-1443-4701-a4e6-0a5789a69eec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:38:19.377376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.360935    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlsl6\" (UniqueName: \"kubernetes.io/projected/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-kube-api-access-jlsl6\") pod \"payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k\" (UID: \"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:38:19.377376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.360978    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4scpq\" (UniqueName: \"kubernetes.io/projected/5be589a6-1443-4701-a4e6-0a5789a69eec-kube-api-access-4scpq\") pod \"payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt\" (UID: \"5be589a6-1443-4701-a4e6-0a5789a69eec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:38:19.377376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.361066    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k\" (UID: \"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:38:19.476314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.468714    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k\" (UID: \"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:38:19.476314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.469680    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be589a6-1443-4701-a4e6-0a5789a69eec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt\" (UID: \"5be589a6-1443-4701-a4e6-0a5789a69eec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:38:19.602152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.602074    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4scpq\" (UniqueName: \"kubernetes.io/projected/5be589a6-1443-4701-a4e6-0a5789a69eec-kube-api-access-4scpq\") pod \"payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt\" (UID: \"5be589a6-1443-4701-a4e6-0a5789a69eec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:38:19.603088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.603058    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:38:19.636108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.636036    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlsl6\" (UniqueName: \"kubernetes.io/projected/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-kube-api-access-jlsl6\") pod \"payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k\" (UID: \"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:38:19.638517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.638484    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8" event={"ID":"d9c26f79-3936-4990-8b0a-3b7e2b32de0e","Type":"ContainerStarted","Data":"fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732"}
Nov 02 01:38:19.638939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.638908    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:38:19.664216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.664163    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8" podStartSLOduration=2.6641470910000002 podStartE2EDuration="2.664147091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:19.662967429 +0000 UTC m=+1404.967900255" watchObservedRunningTime="2026-11-02 01:38:19.664147091 +0000 UTC m=+1404.969079917"
Nov 02 01:38:19.943463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:19.936031    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:38:20.096317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:20.083976    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"]
Nov 02 01:38:20.164700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:20.164522    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4adfa880-dvpc2"]
Nov 02 01:38:20.205362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:20.198494    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"]
Nov 02 01:38:20.513253 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:38:20.513219    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c06bfd_cd8f_49ef_ad57_fc5653a65e7b.slice/crio-85b9cffbd6be7649ee62634ef349dc00749423cd7a974aac89cb286f7ab31fbc WatchSource:0}: Error finding container 85b9cffbd6be7649ee62634ef349dc00749423cd7a974aac89cb286f7ab31fbc: Status 404 returned error can't find the container with id 85b9cffbd6be7649ee62634ef349dc00749423cd7a974aac89cb286f7ab31fbc
Nov 02 01:38:20.516411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:20.516380    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"]
Nov 02 01:38:20.651092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:20.651058    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k" event={"ID":"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b","Type":"ContainerStarted","Data":"85b9cffbd6be7649ee62634ef349dc00749423cd7a974aac89cb286f7ab31fbc"}
Nov 02 01:38:20.655530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:20.655502    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt" event={"ID":"5be589a6-1443-4701-a4e6-0a5789a69eec","Type":"ContainerStarted","Data":"6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a"}
Nov 02 01:38:20.655659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:20.655553    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt" event={"ID":"5be589a6-1443-4701-a4e6-0a5789a69eec","Type":"ContainerStarted","Data":"419e7e772a90b2708dfdbfc34f737d49e49fe35359e2bba8b4cf022f68719c2f"}
Nov 02 01:38:20.655659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:20.655598    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:38:20.679166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:20.679094    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt" podStartSLOduration=1.679072908 podStartE2EDuration="1.679072908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:20.677377813 +0000 UTC m=+1405.982310641" watchObservedRunningTime="2026-11-02 01:38:20.679072908 +0000 UTC m=+1405.984005734"
Nov 02 01:38:21.377934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:21.377895    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825c1938-b9c8-4297-8ad8-d4ebe83ad844" path="/var/lib/kubelet/pods/825c1938-b9c8-4297-8ad8-d4ebe83ad844/volumes"
Nov 02 01:38:21.795632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:21.768332    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k" event={"ID":"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b","Type":"ContainerStarted","Data":"468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0"}
Nov 02 01:38:21.795632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:21.768420    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:38:21.795632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:21.768472    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:38:21.885140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:21.885066    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k" podStartSLOduration=2.88504529 podStartE2EDuration="2.88504529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:21.819158998 +0000 UTC m=+1407.124091825" watchObservedRunningTime="2026-11-02 01:38:21.88504529 +0000 UTC m=+1407.189978116"
Nov 02 01:38:21.950476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:21.950435    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:38:22.035316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:22.035250    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:38:22.041478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:22.037050    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:38:22.540316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:22.539360    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:38:22.709369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:22.708727    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:38:30.657343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:30.657313    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:38:32.810406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:32.810374    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:38:33.826066 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:33.826038    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:38:35.789134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.789101    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"]
Nov 02 01:38:35.795624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.795603    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:35.800703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.800675    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b34fc0-openshift-default-dockercfg-p7dfb\""
Nov 02 01:38:35.805144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.805117    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"]
Nov 02 01:38:35.911433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.911394    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-certs\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:35.911655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.911518    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-istio-token\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:35.911655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.911614    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-envoy\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:35.911839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.911821    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-data\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:35.911916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.911852    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5126a114-46de-4898-8088-c19a89fb51f1-istiod-ca-cert\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:35.911966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.911929    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j58b\" (UniqueName: \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-kube-api-access-5j58b\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:35.912043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.912021    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:35.912093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.912077    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-credential-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:35.912144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:35.912110    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5126a114-46de-4898-8088-c19a89fb51f1-istio-podinfo\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.014691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.014657    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-certs\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.014882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.014720    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-istio-token\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.014882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.014804    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-envoy\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.014992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.014921    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-data\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.014992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.014945    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5126a114-46de-4898-8088-c19a89fb51f1-istiod-ca-cert\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.015170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.014992    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j58b\" (UniqueName: \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-kube-api-access-5j58b\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.015170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.015031    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.015170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.015112    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-certs\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.015170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.015120    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-credential-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.015460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.015175    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5126a114-46de-4898-8088-c19a89fb51f1-istio-podinfo\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.015460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.015354    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-data\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.015639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.015618    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.015832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.015813    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-credential-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.015915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.015868    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5126a114-46de-4898-8088-c19a89fb51f1-istiod-ca-cert\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.017383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.017362    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-envoy\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.017848 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.017832    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5126a114-46de-4898-8088-c19a89fb51f1-istio-podinfo\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.027513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.027494    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-istio-token\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.027734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.027715    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j58b\" (UniqueName: \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-kube-api-access-5j58b\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.115550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.115466    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:36.418953 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:38:36.418903    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5126a114_46de_4898_8088_c19a89fb51f1.slice/crio-725bf46f97aca4038f33f42e2c1f79896f4d8ac07e5b70e2b4f8c1dd0a447db4 WatchSource:0}: Error finding container 725bf46f97aca4038f33f42e2c1f79896f4d8ac07e5b70e2b4f8c1dd0a447db4: Status 404 returned error can't find the container with id 725bf46f97aca4038f33f42e2c1f79896f4d8ac07e5b70e2b4f8c1dd0a447db4
Nov 02 01:38:36.419243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.419196    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"]
Nov 02 01:38:36.422394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.422357    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:38:36.422492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.422428    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:38:36.422492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:36.422457    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:38:37.183197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:37.183145    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" event={"ID":"5126a114-46de-4898-8088-c19a89fb51f1","Type":"ContainerStarted","Data":"69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229"}
Nov 02 01:38:37.183645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:37.183217    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" event={"ID":"5126a114-46de-4898-8088-c19a89fb51f1","Type":"ContainerStarted","Data":"725bf46f97aca4038f33f42e2c1f79896f4d8ac07e5b70e2b4f8c1dd0a447db4"}
Nov 02 01:38:37.211785 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:37.211736    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" podStartSLOduration=2.211720208 podStartE2EDuration="2.211720208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:37.207338037 +0000 UTC m=+1422.512270859" watchObservedRunningTime="2026-11-02 01:38:37.211720208 +0000 UTC m=+1422.516653036"
Nov 02 01:38:38.116646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:38.116603    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:38.150388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:38.132669    4851 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 01:38:38.150388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:38.132736    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" podUID="5126a114-46de-4898-8088-c19a89fb51f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 01:38:39.121311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:39.120603    4851 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 01:38:39.121311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:39.120673    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" podUID="5126a114-46de-4898-8088-c19a89fb51f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 01:38:40.116268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:40.116234    4851 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 01:38:40.116456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:40.116323    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" podUID="5126a114-46de-4898-8088-c19a89fb51f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 01:38:41.116185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:41.116148    4851 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 01:38:41.116742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:41.116210    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" podUID="5126a114-46de-4898-8088-c19a89fb51f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 01:38:42.116820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:42.116787    4851 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 01:38:42.117251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:42.116844    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" podUID="5126a114-46de-4898-8088-c19a89fb51f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 01:38:43.116003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:43.115965    4851 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 02 01:38:43.116235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:43.116035    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" podUID="5126a114-46de-4898-8088-c19a89fb51f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 02 01:38:44.121138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:44.121107    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:44.121617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:44.121604    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:44.122324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:44.122305    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:38:47.658064 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:38:47.658023    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:38:52.217575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.217537    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"]
Nov 02 01:38:52.223995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.223970    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.231145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.231124    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9a4595-openshift-default-dockercfg-trdc7\""
Nov 02 01:38:52.238084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.238058    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"]
Nov 02 01:38:52.407075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.407034    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.413326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.407534    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-certs\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.413639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.413611    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-istio-token\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.413874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.413849    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbtl\" (UniqueName: \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-kube-api-access-vkbtl\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.414255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.414226    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-envoy\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.420481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.420447    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099e6161-f6ee-4eb9-b200-715180f65227-istiod-ca-cert\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.435445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.435399    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099e6161-f6ee-4eb9-b200-715180f65227-istio-podinfo\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.435842 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.435819    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-data\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.436194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.436171    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-credential-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538057    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538269    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-certs\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538323    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-istio-token\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538354    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbtl\" (UniqueName: \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-kube-api-access-vkbtl\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538469    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-envoy\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538509    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538540    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099e6161-f6ee-4eb9-b200-715180f65227-istiod-ca-cert\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538613    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-certs\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538917 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538637    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099e6161-f6ee-4eb9-b200-715180f65227-istio-podinfo\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538917 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538775    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-data\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.538917 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.538816    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-credential-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.539210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.539183    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-credential-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.539397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.539371    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099e6161-f6ee-4eb9-b200-715180f65227-istiod-ca-cert\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.539492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.539471    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-data\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.541413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.541278    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-envoy\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.541543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.541441    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099e6161-f6ee-4eb9-b200-715180f65227-istio-podinfo\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.551225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.551207    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-istio-token\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.551462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.551444    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbtl\" (UniqueName: \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-kube-api-access-vkbtl\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:52.844034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:52.843941    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:53.244316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:53.243748    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"]
Nov 02 01:38:53.278053 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:38:53.278017    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099e6161_f6ee_4eb9_b200_715180f65227.slice/crio-5b2d079aaec005105e6b84bf00da7ffcaec9751a910852a784b363d154117962 WatchSource:0}: Error finding container 5b2d079aaec005105e6b84bf00da7ffcaec9751a910852a784b363d154117962: Status 404 returned error can't find the container with id 5b2d079aaec005105e6b84bf00da7ffcaec9751a910852a784b363d154117962
Nov 02 01:38:53.326315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:53.324155    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:38:53.326315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:53.324268    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:38:53.326315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:53.324341    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:38:53.806628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:53.806591    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" event={"ID":"099e6161-f6ee-4eb9-b200-715180f65227","Type":"ContainerStarted","Data":"2b681c18b956eaa5c0a9db4063765ff4564890e1b9e1ae9726686a31d891e5df"}
Nov 02 01:38:53.806812 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:53.806646    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" event={"ID":"099e6161-f6ee-4eb9-b200-715180f65227","Type":"ContainerStarted","Data":"5b2d079aaec005105e6b84bf00da7ffcaec9751a910852a784b363d154117962"}
Nov 02 01:38:53.842110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:53.842040    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" podStartSLOduration=1.8420208420000002 podStartE2EDuration="1.842020842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:53.833647274 +0000 UTC m=+1439.138580101" watchObservedRunningTime="2026-11-02 01:38:53.842020842 +0000 UTC m=+1439.146953671"
Nov 02 01:38:53.844589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:53.844565    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:38:54.862940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:54.862889    4851 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 01:38:54.863425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:54.862954    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 01:38:55.856315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:55.855643    4851 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 01:38:55.856315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:55.855718    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 01:38:56.856408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:56.852731    4851 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 01:38:56.856408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:56.852804    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 01:38:57.851829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:57.851781    4851 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 01:38:57.854010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:57.853968    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 01:38:58.831922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:58.831873    4851 scope.go:122] "RemoveContainer" containerID="f639cfb58d91a4134f2ddb650c6bcaf96fb7ba291716a4a21cc3bc4d343f1969"
Nov 02 01:38:58.847246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:58.846796    4851 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 01:38:58.847246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:58.846856    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 01:38:58.881839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:58.881815    4851 scope.go:122] "RemoveContainer" containerID="3e4005a4d2c8f1e7f280b68df3bf74ef068fb949efc774ed5d2cf94464ffd59f"
Nov 02 01:38:59.858476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:59.858440    4851 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 01:38:59.859029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:38:59.858520    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 01:39:00.881521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:00.881495    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:39:00.882727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:00.882700    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:39:00.885597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:00.885572    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:39:09.082109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.082053    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:09.156814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.156735    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:39:09.304482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.304424    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:09.597526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.597434    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:39:09.599264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.599234    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:39:09.618859 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.618820    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:09.653326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.639505    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:39:09.718311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.706886    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:09.793769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.768529    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:39:09.926403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.912427    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:09.926403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.912922    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:39:09.994317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:09.969651    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:39:10.048437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.048388    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:39:10.145341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.125519    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:39:10.145341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.133455    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:39:10.174729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.171229    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:10.188594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.188477    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:39:10.336337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.316747    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:10.503230 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.496595    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:39:10.622322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.608715    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:10.622322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.609667    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:39:10.622322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.609883    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:39:10.622322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.609946    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:39:10.637431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.637388    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:10.663499 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.642050    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:10.663886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.663860    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:39:10.702315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.674679    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:39:10.745327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.703028    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:39:10.770374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.707393    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:10.770374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.751185    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:39:10.806487 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.806429    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:10.815320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.811407    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:39:10.974354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.954791    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:39:10.993415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:10.993370    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:39:11.012772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.005981    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:11.085383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.063716    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:11.167424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.167339    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:39:11.185325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.181059    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:11.462519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.444211    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:39:11.462519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.444306    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:39:11.462519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.444350    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:39:11.462519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.444394    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:39:11.511329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.507402    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:11.618126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.618080    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:11.728416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.619327    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:39:11.760562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.760499    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:11.817316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.790931    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:39:12.347435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.812840    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:39:12.347435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:11.987719    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:39:12.347435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.020504    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:39:12.347435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.039882    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:12.347435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.055360    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:12.347435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.133384    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:12.347435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.161082    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:12.409320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.394439    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:39:12.409320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.395505    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:39:12.409320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.395579    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:39:12.709318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.692545    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:39:12.763485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.763422    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:12.977407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:12.975438    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:39:13.075582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.035524    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl" podUID="437397c1-b8b9-4a7a-90db-17411343e18e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:13.094421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.062335    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:13.098313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.062478    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:39:13.123395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.070265    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:39:13.123630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.108715    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:13.137484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.137437    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:39:13.161495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.161445    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:13.261425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.261158    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:39:13.270925 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.267359    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded"
Nov 02 01:39:13.393644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.393598    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:13.421367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.395927    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:39:13.838076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:13.829507    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:39:14.171712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.171379    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:14.179652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.179323    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:39:14.211451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.211385    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded"
Nov 02 01:39:14.549256 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:39:14.522335    4851 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.495s"
Nov 02 01:39:14.603441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.603413    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:39:14.627945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.217631    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded"
Nov 02 01:39:14.674070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.266792    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:14.675758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.361624    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9" podUID="d46b2d7a-fabe-4466-818c-9fc511dbd10e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/ready\": context deadline exceeded"
Nov 02 01:39:14.676366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.370399    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:39:14.681314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.372103    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Nov 02 01:39:14.786938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.418395    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:14.862370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.434673    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:39:14.862370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.507616    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:39:14.862370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.857228    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:39:14.862370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.217566    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:14.941680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.591109    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:14.941680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.923363    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:15.172093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.172047    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:15.243945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:14.217605    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr" podUID="d52b9b8d-07d7-49d9-b7e0-92cd937b26c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Nov 02 01:39:15.246036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.176556    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:39:15.269320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.181073    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:15.269320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.181690    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:15.288446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.288394    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded"
Nov 02 01:39:15.288760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.288734    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:39:15.325578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.325534    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:39:15.556177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.553712    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded"
Nov 02 01:39:15.688340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.679622    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:15.691387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.689352    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:39:15.860070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.859935    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:15.886348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.860334    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:39:15.886348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.860385    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:39:15.993319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:15.966676    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:39:16.023961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.022552    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:39:16.370328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.313018    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:16.370328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.314559    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:39:16.372469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.372404    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:39:16.405341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.402671    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:39:16.494615 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:39:16.494575    4851 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.248s"
Nov 02 01:39:16.494946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.494925    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:39:16.495359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.495342    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:39:16.495536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.495522    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:39:16.495757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.495743    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:39:16.548316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.524062    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:39:16.555761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.555716    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:16.870251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.870199    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:16.889317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.872126    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Nov 02 01:39:16.889893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.818426    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:16.948948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.948903    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:16.951894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:16.951825    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:39:17.099318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:17.078554    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:17.349964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:17.338431    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded"
Nov 02 01:39:17.414486 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:17.385002    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:17.500482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:17.500434    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 02 01:39:17.626416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:17.626308    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:39:17.685541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:17.685498    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:17.777460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:17.777403    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:17.837400 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:17.832314    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded"
Nov 02 01:39:17.907833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:17.904815    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:17.907833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:17.905006    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:39:18.007352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.006724    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9" podUID="d46b2d7a-fabe-4466-818c-9fc511dbd10e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:18.007352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.007127    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:18.007630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.007469    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:39:18.026413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.007734    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr" podUID="d52b9b8d-07d7-49d9-b7e0-92cd937b26c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Nov 02 01:39:18.088443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.088398    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:18.157173 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:39:18.132213    4851 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.18s"
Nov 02 01:39:18.157173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.148674    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:39:18.157173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.149038    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:39:18.176158 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.171844    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:18.180315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.176499    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:39:18.444487 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.414006    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:18.454929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.454886    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:18.560832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.552470    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:39:18.560832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.555810    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:39:18.608038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.608001    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:18.650659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.650613    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:39:18.687755 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.687722    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:39:18.706974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.695051    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:39:18.716764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.712159    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:39:18.718070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.718049    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:39:18.720404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.720371    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:18.721452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.721432    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:39:18.754495 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:39:18.725749    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:39:18.754495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.726567    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:39:18.758508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.755064    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:39:18.831541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:18.828535    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:39:20.585245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.585189    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"]
Nov 02 01:39:20.610085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.609445    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"]
Nov 02 01:39:20.610085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.609642    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:39:20.614660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.614633    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"]
Nov 02 01:39:20.630821 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.630790    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-14b63f\""
Nov 02 01:39:20.631015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.630926    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-14b63f-dockercfg-jpr4j\""
Nov 02 01:39:20.638139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.638112    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"]
Nov 02 01:39:20.638259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.638228    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:39:20.693573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.693443    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9241dd32-519f-47a0-b614-2211204c6e8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7\" (UID: \"9241dd32-519f-47a0-b614-2211204c6e8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:39:20.693573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.693544    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq8zx\" (UniqueName: \"kubernetes.io/projected/e7f9e8df-23b6-478d-b95b-00a6989afed7-kube-api-access-qq8zx\") pod \"payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn\" (UID: \"e7f9e8df-23b6-478d-b95b-00a6989afed7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:39:20.693907 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.693681    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9gv\" (UniqueName: \"kubernetes.io/projected/9241dd32-519f-47a0-b614-2211204c6e8b-kube-api-access-nw9gv\") pod \"payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7\" (UID: \"9241dd32-519f-47a0-b614-2211204c6e8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:39:20.694002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.693908    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f9e8df-23b6-478d-b95b-00a6989afed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn\" (UID: \"e7f9e8df-23b6-478d-b95b-00a6989afed7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:39:20.891365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.843267    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9241dd32-519f-47a0-b614-2211204c6e8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7\" (UID: \"9241dd32-519f-47a0-b614-2211204c6e8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:39:20.891365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.843380    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8zx\" (UniqueName: \"kubernetes.io/projected/e7f9e8df-23b6-478d-b95b-00a6989afed7-kube-api-access-qq8zx\") pod \"payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn\" (UID: \"e7f9e8df-23b6-478d-b95b-00a6989afed7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:39:20.891365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.843455    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9gv\" (UniqueName: \"kubernetes.io/projected/9241dd32-519f-47a0-b614-2211204c6e8b-kube-api-access-nw9gv\") pod \"payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7\" (UID: \"9241dd32-519f-47a0-b614-2211204c6e8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:39:20.891365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.859324    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9241dd32-519f-47a0-b614-2211204c6e8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7\" (UID: \"9241dd32-519f-47a0-b614-2211204c6e8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:39:20.909323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.893169    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f9e8df-23b6-478d-b95b-00a6989afed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn\" (UID: \"e7f9e8df-23b6-478d-b95b-00a6989afed7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:39:20.909323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:20.894348    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f9e8df-23b6-478d-b95b-00a6989afed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn\" (UID: \"e7f9e8df-23b6-478d-b95b-00a6989afed7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:39:21.161425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:21.161391    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9gv\" (UniqueName: \"kubernetes.io/projected/9241dd32-519f-47a0-b614-2211204c6e8b-kube-api-access-nw9gv\") pod \"payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7\" (UID: \"9241dd32-519f-47a0-b614-2211204c6e8b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:39:21.164022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:21.163984    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq8zx\" (UniqueName: \"kubernetes.io/projected/e7f9e8df-23b6-478d-b95b-00a6989afed7-kube-api-access-qq8zx\") pod \"payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn\" (UID: \"e7f9e8df-23b6-478d-b95b-00a6989afed7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:39:21.264758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:21.264585    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:39:21.266741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:21.266634    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:39:21.818038 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:39:21.818004    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f9e8df_23b6_478d_b95b_00a6989afed7.slice/crio-de48d6d16fef3ed99373e91b5ed10eac9f78f8f0ab1bd96e3315b5d48ea77fee WatchSource:0}: Error finding container de48d6d16fef3ed99373e91b5ed10eac9f78f8f0ab1bd96e3315b5d48ea77fee: Status 404 returned error can't find the container with id de48d6d16fef3ed99373e91b5ed10eac9f78f8f0ab1bd96e3315b5d48ea77fee
Nov 02 01:39:21.823679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:21.822376    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"]
Nov 02 01:39:21.838123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:21.833376    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"]
Nov 02 01:39:22.375189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:22.370319    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn" event={"ID":"e7f9e8df-23b6-478d-b95b-00a6989afed7","Type":"ContainerStarted","Data":"4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8"}
Nov 02 01:39:22.375189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:22.370395    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn" event={"ID":"e7f9e8df-23b6-478d-b95b-00a6989afed7","Type":"ContainerStarted","Data":"de48d6d16fef3ed99373e91b5ed10eac9f78f8f0ab1bd96e3315b5d48ea77fee"}
Nov 02 01:39:22.375189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:22.371137    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:39:22.393873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:22.393825    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7" event={"ID":"9241dd32-519f-47a0-b614-2211204c6e8b","Type":"ContainerStarted","Data":"440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc"}
Nov 02 01:39:22.394300 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:22.394217    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7" event={"ID":"9241dd32-519f-47a0-b614-2211204c6e8b","Type":"ContainerStarted","Data":"dd9393d79e91d769814ed7ec43315ffd6528178d575239c53212382fb29bba7b"}
Nov 02 01:39:22.398512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:22.398457    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:39:22.762422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:22.746147    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn" podStartSLOduration=2.746124927 podStartE2EDuration="2.746124927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:22.391685943 +0000 UTC m=+1467.696618770" watchObservedRunningTime="2026-11-02 01:39:22.746124927 +0000 UTC m=+1468.051057754"
Nov 02 01:39:23.033205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:23.028426    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7" podStartSLOduration=3.028409083 podStartE2EDuration="3.028409083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:22.711863332 +0000 UTC m=+1468.016796183" watchObservedRunningTime="2026-11-02 01:39:23.028409083 +0000 UTC m=+1468.333341911"
Nov 02 01:39:26.761365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:26.705461    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:26.796984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:26.796927    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:27.344461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:27.344421    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:27.344660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:27.344498    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:29.945038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:29.944990    4851 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-cqpmz container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": context deadline exceeded" start-of-body=
Nov 02 01:39:29.970621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:29.945078    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": context deadline exceeded"
Nov 02 01:39:30.090227 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:39:29.985529    4851 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.065s"
Nov 02 01:39:30.745467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:30.745418    4851 patch_prober.go:29] interesting pod/jobset-controller-manager-5b9f7dd4f5-84bht container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.30:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:30.766582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:30.745483    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-5b9f7dd4f5-84bht" podUID="8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.30:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:31.214460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:31.214424    4851 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-cqpmz container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.13:8443/healthz\": context deadline exceeded" start-of-body=
Nov 02 01:39:32.628865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:31.214480    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/healthz\": context deadline exceeded"
Nov 02 01:39:32.628865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:31.319425    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl" podUID="437397c1-b8b9-4a7a-90db-17411343e18e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:32.628865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:31.438385    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:32.628865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:31.438431    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:32.628865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:31.948875    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl" podUID="437397c1-b8b9-4a7a-90db-17411343e18e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Nov 02 01:39:41.935877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:41.770497    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr" podUID="d52b9b8d-07d7-49d9-b7e0-92cd937b26c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:41.935877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:41.770825    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:41.935877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:41.770922    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl" podUID="437397c1-b8b9-4a7a-90db-17411343e18e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded"
Nov 02 01:39:41.935877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:41.771365    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:41.935877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:41.771416    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 01:39:41.935877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:41.797747    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4" podUID="21310443-4ec4-40ae-91da-65292ae655f9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.183:50052\" within 1s: context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.064568    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz" podUID="5eafdd80-7407-40d6-9579-cedb558b6411" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.193:50052\" within 1s: context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:39:42.064711    4851 logging.go:55] [core] [Channel #3211 SubChannel #3214]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.193:50052", ServerName: "10.132.0.193:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.193:50052: operation was canceled"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.083190    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf" podUID="e377a587-5a3a-46d6-a95e-b2a0decdf9c3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.170:50052\" within 1s: context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:39:42.092818    4851 logging.go:55] [core] [Channel #3198 SubChannel #3200]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.183:50052", ServerName: "10.132.0.183:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.183:50052: operation was canceled"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.093880    4851 patch_prober.go:29] interesting pod/router-default-5c986fbdbf-f69k6 container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.21:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.093921    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-5c986fbdbf-f69k6" podUID="cfeff66f-98e2-422f-a3be-cfc9f1d7a829" containerName="router" probeResult="failure" output="Get \"http://10.132.0.21:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.094146    4851 patch_prober.go:29] interesting pod/dns-default-7zmf4 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.16:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.282038    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-7zmf4" podUID="1be28aae-f92c-4eb1-b063-f84b46967f3b" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.16:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.280064    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn" podUID="e7f9e8df-23b6-478d-b95b-00a6989afed7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.210:50052\" within 1s: context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.273795    4851 patch_prober.go:29] interesting pod/dns-default-7zmf4 container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.132.0.16:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.283034    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-7zmf4" podUID="1be28aae-f92c-4eb1-b063-f84b46967f3b" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.16:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:39:42.280483    4851 logging.go:55] [core] [Channel #3197 SubChannel #3223]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.210:50052", ServerName: "10.132.0.210:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.210:50052: operation was canceled"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:39:42.281449    4851 logging.go:55] [core] [Channel #3210 SubChannel #3213]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.170:50052", ServerName: "10.132.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.170:50052: operation was canceled"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.283824    4851 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.933457285s: [/var/lib/containers/storage/overlay/1f5be7ec9c274fbe7b332d38fe0162aa6dbd5ff16eba135feee425d369d7b7e0/diff /var/log/pods/openshift-ingress_e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69_39a26fed-b9e3-4085-9f05-d33b5532a9df/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.302514    4851 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.448010875s: [/var/lib/containers/storage/overlay/d0df459f841f92dff46e35bea35a35deeb0d5998d63940d4c7ad631e01d3ecee/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-ns4lc_0de30efc-e5bc-428c-ba8d-6b7967e29473/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.637953    4851 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.287604862s: [/var/lib/containers/storage/overlay/db60c45f8a6de9b5392dbe2f310eb3ec02788167d7a5f7b5a2c96a35a6006c26/diff /var/log/pods/openshift-ingress_e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs_3f4ce5ae-b0b2-4ba7-8c04-672afae643ed/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.645984    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.646456    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.646809    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.767853    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.767889    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.768044    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.768434    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.770870    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.770899    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr" podUID="d52b9b8d-07d7-49d9-b7e0-92cd937b26c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.771102    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl" podUID="437397c1-b8b9-4a7a-90db-17411343e18e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.771227    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.771391    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.771584    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.771733    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.771860    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.774069    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.780269    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.787431    4851 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-cqpmz container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.787473    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.808429    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.808438    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh" podUID="1e5b2266-6f15-44c8-b44c-e6288fae68ba" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.808475    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.808429    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.808517    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.810422    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.810466    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.811382    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.811414    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.812453    4851 patch_prober.go:29] interesting pod/router-default-5c986fbdbf-f69k6 container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.21:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.812480    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-5c986fbdbf-f69k6" podUID="cfeff66f-98e2-422f-a3be-cfc9f1d7a829" containerName="router" probeResult="failure" output="Get \"http://10.132.0.21:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.813414    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx" podUID="90d90625-4204-4fec-8fb1-1237f358bd6a" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.813438    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k" podUID="01c06bfd-cd8f-49ef-ad57-fc5653a65e7b" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.813448    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9vj42" podUID="b5d69b63-2892-44fd-a2ba-777e1278ecc5" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.815404    4851 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.815432    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.819392    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7" podUID="9241dd32-519f-47a0-b614-2211204c6e8b" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.822439    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn" podUID="592ace31-a3ce-42ef-b168-64712a32a593" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.825443    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.825474    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.832435    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn" podUID="592ace31-a3ce-42ef-b168-64712a32a593" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.833433    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.833465    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.836411    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.836451    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.837433    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m" podUID="7d5a5b02-0b4a-4b54-a8e5-417b50f91a06" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.838473    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.838503    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9" podUID="70565ca1-c120-4b77-b307-a16604644193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.947160    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9" podUID="d46b2d7a-fabe-4466-818c-9fc511dbd10e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:42.947319    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.005481    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln" podUID="c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.026169    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl" podUID="437397c1-b8b9-4a7a-90db-17411343e18e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.116666    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8" podUID="d9c26f79-3936-4990-8b0a-3b7e2b32de0e" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.131419    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5" podUID="fe26ff80-ecf0-4541-adc0-56877c9c009d" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.140537    4851 patch_prober.go:29] interesting pod/router-default-5c986fbdbf-f69k6 container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.21:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.140595    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-5c986fbdbf-f69k6" podUID="cfeff66f-98e2-422f-a3be-cfc9f1d7a829" containerName="router" probeResult="failure" output="Get \"http://10.132.0.21:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.265708    4851 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.481315833s: [/var/lib/containers/storage/overlay/58d33df61256ef9aae5a59977ee0af1e84abe808016d7e58f9c1413e82c615b8/diff /var/log/pods/openshift-ingress_e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q_5126a114-46de-4898-8088-c19a89fb51f1/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.283616    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.284029    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.299632    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.300553    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.300850    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.301113    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-srgl9" podUID="d46b2d7a-fabe-4466-818c-9fc511dbd10e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.314634    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt" podUID="5be589a6-1443-4701-a4e6-0a5789a69eec" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.314805    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.314870    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr" podUID="d52b9b8d-07d7-49d9-b7e0-92cd937b26c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.317302    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.318376    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:48.862456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.318506    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.318629    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.318657    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.324369    4851 patch_prober.go:29] interesting pod/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.324412    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.324845    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh" podUID="fa4d91d9-ce2f-48f6-a164-720a5efcab51" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.325211    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf" podUID="b3daa532-ae71-4a64-a684-59d4fbb22088" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.325318    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.325344    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.325430    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8" podUID="c6ac98d6-c879-4606-81a4-b791e4af5b3f" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.325513    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.325538    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.327236    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns" podUID="4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.327631    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.327667    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.327960    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh" podUID="bb9669fc-e378-4a4b-aea5-ea97c9b71882" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.328142    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds" podUID="578eb675-6e20-4ed1-a546-a0bb01f3f899" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.328235    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h" podUID="5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.328350    4851 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-cqpmz container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.13:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.328390    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-cqpmz" podUID="b395656d-d1a0-4cea-9b61-41788ab06732" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.329460    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z" podUID="0f7739d3-8fb8-453e-8af5-345d8bd842e4" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.330807    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-ns4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.330840    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podUID="0de30efc-e5bc-428c-ba8d-6b7967e29473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.338380    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st" podUID="1e34ceb8-6e96-4404-bfd7-728e16738621" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.338482    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p" podUID="14b4b0ad-6f18-410d-b545-db6f1ecd3141" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.339367    4851 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.339416    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.341374    4851 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.341414    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.360420    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc" podUID="41550ac8-635a-4162-bbce-4145804093be" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.366421    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5" podUID="2ec4c675-7706-4edf-808a-a542b0986037" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.631131    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.642484    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.646727    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.646731    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.646975    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.651565    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.682397    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.682429    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.768356    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.768361    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.768519    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.771779    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.771797    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.771820    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr" podUID="d52b9b8d-07d7-49d9-b7e0-92cd937b26c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.771857    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.771825    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.855771    4851 patch_prober.go:29] interesting pod/router-default-5c986fbdbf-f69k6 container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.21:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.855843    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-5c986fbdbf-f69k6" podUID="cfeff66f-98e2-422f-a3be-cfc9f1d7a829" containerName="router" probeResult="failure" output="Get \"http://10.132.0.21:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:43.861344    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:44.367406    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh" podUID="fa4d91d9-ce2f-48f6-a164-720a5efcab51" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:44.821380    4851 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-dw7wh container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:44.821419    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh" podUID="5f29f286-e8a8-41e8-b0b7-c18dacdbdd83" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:45.337388    4851 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-dw7wh container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:45.337502    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh" podUID="5f29f286-e8a8-41e8-b0b7-c18dacdbdd83" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:46.438402    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:46.438453    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:39:46.733089    4851 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="4.633s"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:46.733192    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:46.733665    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"]
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:46.733735    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:46.921267    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.296559    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.296609    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jgzjl"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.296667    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.296709    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.296746    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.296767    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da359c04-vhznb"]
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.296840    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.296894    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.296953    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.296991    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.297026    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.297059    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.297090    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.297122    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.297147    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.297175    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.297191    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.297714    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.302624    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.302672    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.302692    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:47.302732    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:48.477482    4851 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-dw7wh container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:48.477547    4851 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dw7wh" podUID="5f29f286-e8a8-41e8-b0b7-c18dacdbdd83" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:48.868808    4851 generic.go:356] "Generic (PLEG): container finished" podID="fa4d91d9-ce2f-48f6-a164-720a5efcab51" containerID="08629e704e0ea18fb89e80705fff2b6078e21fde4947a6b5093ee1b5207eb8bd" exitCode=1
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:48.923726    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aadab6a4-b16f-4ece-a644-480cfe0dd89b" path="/var/lib/kubelet/pods/aadab6a4-b16f-4ece-a644-480cfe0dd89b/volumes"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:48.951834    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh" event={"ID":"fa4d91d9-ce2f-48f6-a164-720a5efcab51","Type":"ContainerDied","Data":"08629e704e0ea18fb89e80705fff2b6078e21fde4947a6b5093ee1b5207eb8bd"}
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:39:48.955482    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:48.956774    4851 scope.go:122] "RemoveContainer" containerID="08629e704e0ea18fb89e80705fff2b6078e21fde4947a6b5093ee1b5207eb8bd"
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:48.957536    4851 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:39:49.239107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:49.177347    4851 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:39:50.682602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:50.682575    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 01:39:52.922023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:52.921996    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh" event={"ID":"fa4d91d9-ce2f-48f6-a164-720a5efcab51","Type":"ContainerStarted","Data":"27615c25f375e87e2f4bc5e6fdec27b885f9eb80e07926be3fdd4c15aea9d68f"}
Nov 02 01:39:52.922548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:52.922133    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:39:55.852095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:55.852061    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:39:55.852095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:55.852061    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:39:56.756490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:56.756454    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:56.770855 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:56.756537    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:57.343509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:57.343458    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:39:57.378041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:57.343536    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:39:59.252261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:39:59.252223    4851 scope.go:122] "RemoveContainer" containerID="0e708bdd297115b71eadf4592f08cf159a43d34ec9136198113c45cf4291572d"
Nov 02 01:40:00.145152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:00.106753    4851 scope.go:122] "RemoveContainer" containerID="ac5d528b4ea64d879ab4efcebb3ede877a995cffe1cf7436acb47a6c832a14a4"
Nov 02 01:40:01.438438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:01.438402    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:01.438933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:01.438488    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:03.933331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:03.933267    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-8445457bbd-cjjmh"
Nov 02 01:40:11.732455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:11.732413    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:11.733098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:11.732484    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:11.733098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:11.732584    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:40:12.353556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:12.353512    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:12.450806 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:12.353836    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:12.496153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:12.475331    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:40:12.774475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:12.774431    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:12.775002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:12.774500    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:12.957458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:12.957418    4851 patch_prober.go:29] interesting pod/router-default-5c986fbdbf-f69k6 container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.21:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:12.957730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:12.957496    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-5c986fbdbf-f69k6" podUID="cfeff66f-98e2-422f-a3be-cfc9f1d7a829" containerName="router" probeResult="failure" output="Get \"http://10.132.0.21:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:13.517459 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:13.517408    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:13.576899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:13.517494    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:16.438444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:16.438406    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:16.528620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:16.438459    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:16.528620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:16.438544    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:40:17.481495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:17.481455    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:17.481953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:17.481522    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:21.812351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:21.811748    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:21.956556 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:40:21.956500    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:40:27.775478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:27.775433    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:28.085368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:27.775522    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:28.845652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:28.843866    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:28.845652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:28.843953    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:31.136891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:31.136795    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"]
Nov 02 01:40:31.161555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:31.145007    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f73a317-gnn6s"]
Nov 02 01:40:32.480453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:32.480409    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:40:32.480880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:32.480496    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:40:33.009660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:33.009619    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67176ea1-a6c8-4bbb-9cfc-457f12f1c645" path="/var/lib/kubelet/pods/67176ea1-a6c8-4bbb-9cfc-457f12f1c645/volumes"
Nov 02 01:40:33.024801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:33.012600    4851 trace.go:236] Trace[1007618136]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-phmfn" (02-Nov-2026 01:40:26.546) (total time: 6466ms):
Nov 02 01:40:33.024801 ip-10-0-140-23 kubenswrapper[4851]: Trace[1007618136]: [6.466204843s] [6.466204843s] END
Nov 02 01:40:37.598463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.598368    4851 generic.go:356] "Generic (PLEG): container finished" podID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerID="f1769e89b2b8bab680c9f8909458a125a26c2926c597f4cb936a09869cf1c68c" exitCode=0
Nov 02 01:40:37.598463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.598403    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" event={"ID":"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b","Type":"ContainerDied","Data":"f1769e89b2b8bab680c9f8909458a125a26c2926c597f4cb936a09869cf1c68c"}
Nov 02 01:40:37.603197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.600923    4851 generic.go:356] "Generic (PLEG): container finished" podID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerID="f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85" exitCode=0
Nov 02 01:40:37.603197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.601003    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" event={"ID":"548143a8-0c95-41f1-9c60-857747c3c4dd","Type":"ContainerDied","Data":"f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85"}
Nov 02 01:40:37.611593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.605412    4851 scope.go:122] "RemoveContainer" containerID="f1769e89b2b8bab680c9f8909458a125a26c2926c597f4cb936a09869cf1c68c"
Nov 02 01:40:37.611593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.605535    4851 scope.go:122] "RemoveContainer" containerID="f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85"
Nov 02 01:40:38.118439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.631190    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:38.118439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.631376    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:38.118439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.631482    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:38.118439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.632815    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:38.118439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.632965    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:38.118439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:37.633067    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:39.625430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:39.625393    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" event={"ID":"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b","Type":"ContainerStarted","Data":"bfb8c8f67febbeef3182da3ccd1c2e5089da76499b0bd9b3c72b0eec55d3cea4"}
Nov 02 01:40:39.629347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:39.629325    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" event={"ID":"548143a8-0c95-41f1-9c60-857747c3c4dd","Type":"ContainerStarted","Data":"e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e"}
Nov 02 01:40:39.631264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:39.631238    4851 generic.go:356] "Generic (PLEG): container finished" podID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerID="d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df" exitCode=0
Nov 02 01:40:39.631423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:39.631344    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" event={"ID":"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead","Type":"ContainerDied","Data":"d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df"}
Nov 02 01:40:39.631726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:39.631708    4851 scope.go:122] "RemoveContainer" containerID="d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df"
Nov 02 01:40:39.633815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:39.633788    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:39.633991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:39.633872    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:39.633991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:39.633922    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:39.663198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:39.663166    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:40:40.491319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:40.487522    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:40:40.491319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:40.487898    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 01:40:40.491319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:40.487946    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 01:40:40.697049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:40.693656    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 01:40:40.706527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:40.706482    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 01:40:40.759457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:40.759358    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" event={"ID":"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead","Type":"ContainerStarted","Data":"3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda"}
Nov 02 01:40:41.452317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:41.451969    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:40:41.468319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:41.455351    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 01:40:41.468319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:41.455408    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 01:40:41.488319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:41.487732    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 01:40:41.488319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:41.487793    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 01:40:41.672315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:41.668102    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 01:40:41.672315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:41.668220    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 01:40:41.739492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:41.739396    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:40:42.451314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:42.450902    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 01:40:42.451314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:42.450983    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 01:40:42.491318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:42.481409    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:40:42.491318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:42.491030    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 01:40:42.491318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:42.491085    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 01:40:42.682318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:42.681373    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 01:40:42.682318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:42.681446    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 01:40:43.450325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:43.449019    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 01:40:43.450325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:43.449079    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 01:40:43.493317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:43.492551    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 01:40:43.493317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:43.492614    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 01:40:43.670144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:43.670110    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 01:40:43.670464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:43.670435    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 01:40:44.449701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:44.449659    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 01:40:44.450036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:44.450008    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 01:40:44.490878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:44.490843    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 01:40:44.492325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:44.491472    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 01:40:44.669318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:44.666727    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 01:40:44.669318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:44.666802    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 01:40:45.451317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:45.449816    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 01:40:45.451317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:45.449879    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 01:40:45.501323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:45.493463    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 01:40:45.501323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:45.493523    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 01:40:45.670316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:45.663868    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 01:40:45.670316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:45.663927    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 01:40:46.442025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:46.441989    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:40:46.473343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:46.465220    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 01:40:46.473563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:46.473414    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 01:40:46.536512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:46.536442    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 01:40:46.537048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:46.536542    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 01:40:46.742656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:46.742548    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 01:40:46.742959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:46.742928    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 01:40:47.448993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:47.448948    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 01:40:47.449263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:47.449034    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 01:40:47.492776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:47.492740    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:40:47.495195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:47.495168    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:40:47.695361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:47.693929    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:40:47.696776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:47.696746    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:40:47.705639 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:40:47.705568    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:40:48.454722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:48.454696    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:40:48.455981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:40:48.455959    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:41:01.312391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:41:01.312356    4851 scope.go:122] "RemoveContainer" containerID="9dd27fe788844e2eabfc4e5673fefb2308156273f245539ec35c33adf6d2c88e"
Nov 02 01:41:01.328514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:41:01.328486    4851 scope.go:122] "RemoveContainer" containerID="10f742f0710f4b23c9c9b79b8b38ebdbbaad1c0913a095e422bb55578a70cedb"
Nov 02 01:41:01.341527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:41:01.341508    4851 scope.go:122] "RemoveContainer" containerID="2d82e41dcd9e8a80458b853cda2fbcafda772d84bfae81a1410f36da7ea56ad6"
Nov 02 01:41:17.658532 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:41:17.658446    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:41:47.686819 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:41:47.686772    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:41:58.439670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:41:58.439634    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"]
Nov 02 01:41:58.440155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:41:58.439863    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy" containerID="cri-o://2d458e5ff27410e3e425aef91593ccbeb56a2ef8d87aee2aebbe565781de2972" gracePeriod=30
Nov 02 01:42:01.390151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:01.390117    4851 scope.go:122] "RemoveContainer" containerID="9b9c2cfa9f7c8bbd1003c70d1479dc7cc6daccc9cbbae853a96ecc8ac0822ad3"
Nov 02 01:42:01.403851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:01.402892    4851 scope.go:122] "RemoveContainer" containerID="e40daa9ea8e31b04b1b1eee245187dcf52442d9f47baec929cf9e484e5d364d3"
Nov 02 01:42:03.837866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:03.837781    4851 generic.go:356] "Generic (PLEG): container finished" podID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerID="2d458e5ff27410e3e425aef91593ccbeb56a2ef8d87aee2aebbe565781de2972" exitCode=0
Nov 02 01:42:03.837866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:03.837838    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" event={"ID":"7cb94e95-465c-49e8-9ad5-dc74fa3283bc","Type":"ContainerDied","Data":"2d458e5ff27410e3e425aef91593ccbeb56a2ef8d87aee2aebbe565781de2972"}
Nov 02 01:42:04.243605 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.243578    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:42:04.280999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.280971    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"]
Nov 02 01:42:04.281928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.281909    4851 cpu_manager.go:414] "containerMap: removing container" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy"
Nov 02 01:42:04.281928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.281928    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy"
Nov 02 01:42:04.282120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.282109    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" containerName="istio-proxy"
Nov 02 01:42:04.286318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.286281    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.315429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.315401    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"]
Nov 02 01:42:04.342662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.342624    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.344424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.342779    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96fth\" (UniqueName: \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-kube-api-access-96fth\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.344424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.342977    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.344424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.343022    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca405bb-787c-44aa-83ae-cec31331ea1c-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.344424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.343236    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.344424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.343277    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.344424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.343356    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.344424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.343384    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.344424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.343542    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.444588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.444558    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-token\" (UniqueName: \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-token\") pod \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") "
Nov 02 01:42:04.444588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.444593    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-certs\") pod \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") "
Nov 02 01:42:04.444827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.444630    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-credential-socket\") pod \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") "
Nov 02 01:42:04.444827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.444674    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-data\") pod \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") "
Nov 02 01:42:04.444827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.444715    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-podinfo\") pod \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") "
Nov 02 01:42:04.444827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.444777    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istiod-ca-cert\") pod \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") "
Nov 02 01:42:04.444965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.444840    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-kube-api-access-twhzz\" (UniqueName: \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-kube-api-access-twhzz\") pod \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") "
Nov 02 01:42:04.444965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.444912    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-socket\") pod \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") "
Nov 02 01:42:04.444965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.444946    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-envoy\") pod \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\" (UID: \"7cb94e95-465c-49e8-9ad5-dc74fa3283bc\") "
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445280    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96fth\" (UniqueName: \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-kube-api-access-96fth\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445373    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445400    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca405bb-787c-44aa-83ae-cec31331ea1c-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445494    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-certs" pod "7cb94e95-465c-49e8-9ad5-dc74fa3283bc" (UID: "7cb94e95-465c-49e8-9ad5-dc74fa3283bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445625    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445603    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-credential-socket" pod "7cb94e95-465c-49e8-9ad5-dc74fa3283bc" (UID: "7cb94e95-465c-49e8-9ad5-dc74fa3283bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445653    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-data" pod "7cb94e95-465c-49e8-9ad5-dc74fa3283bc" (UID: "7cb94e95-465c-49e8-9ad5-dc74fa3283bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445682    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445709    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445730    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istiod-ca-cert" pod "7cb94e95-465c-49e8-9ad5-dc74fa3283bc" (UID: "7cb94e95-465c-49e8-9ad5-dc74fa3283bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445738    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.445911    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-socket" pod "7cb94e95-465c-49e8-9ad5-dc74fa3283bc" (UID: "7cb94e95-465c-49e8-9ad5-dc74fa3283bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446028    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446059    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446191    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.446347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446262    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca405bb-787c-44aa-83ae-cec31331ea1c-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.449011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446360    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.449011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446651    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.449011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446798    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.449011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446810    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.449011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446819    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.449011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446828    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.449011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446836    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.449011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.446903    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.455055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.455022    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-podinfo" pod "7cb94e95-465c-49e8-9ad5-dc74fa3283bc" (UID: "7cb94e95-465c-49e8-9ad5-dc74fa3283bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:42:04.455055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.455044    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-kube-api-access-twhzz" pod "7cb94e95-465c-49e8-9ad5-dc74fa3283bc" (UID: "7cb94e95-465c-49e8-9ad5-dc74fa3283bc"). InnerVolumeSpecName "kube-api-access-twhzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:04.457073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.455147    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.457073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.455269    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.457073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.455312    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-envoy" pod "7cb94e95-465c-49e8-9ad5-dc74fa3283bc" (UID: "7cb94e95-465c-49e8-9ad5-dc74fa3283bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:04.457073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.455610    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-token" pod "7cb94e95-465c-49e8-9ad5-dc74fa3283bc" (UID: "7cb94e95-465c-49e8-9ad5-dc74fa3283bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:04.458476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.458457    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96fth\" (UniqueName: \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-kube-api-access-96fth\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.458655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.458635    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.548489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.548462    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.548489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.548488    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.548489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.548498    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.549515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.548508    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twhzz\" (UniqueName: \"kubernetes.io/projected/7cb94e95-465c-49e8-9ad5-dc74fa3283bc-kube-api-access-twhzz\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:04.608427 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.608394    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:04.871783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.871762    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"
Nov 02 01:42:04.886730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.886679    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb" event={"ID":"7cb94e95-465c-49e8-9ad5-dc74fa3283bc","Type":"ContainerDied","Data":"0d280970273f9dff1c40e4228ae00df818534be579b30de5925581eaec2a23c2"}
Nov 02 01:42:04.886861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.886734    4851 scope.go:122] "RemoveContainer" containerID="2d458e5ff27410e3e425aef91593ccbeb56a2ef8d87aee2aebbe565781de2972"
Nov 02 01:42:04.906074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.906044    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"]
Nov 02 01:42:04.912577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.912550    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-8x5gb"]
Nov 02 01:42:04.957176 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.957151    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"]
Nov 02 01:42:04.960223 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:42:04.960185    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca405bb_787c_44aa_83ae_cec31331ea1c.slice/crio-4a4bfe3c4535e87f566a362bfddc74a182c7534faef0ee391bcbeeedffe042fb WatchSource:0}: Error finding container 4a4bfe3c4535e87f566a362bfddc74a182c7534faef0ee391bcbeeedffe042fb: Status 404 returned error can't find the container with id 4a4bfe3c4535e87f566a362bfddc74a182c7534faef0ee391bcbeeedffe042fb
Nov 02 01:42:04.963478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.963446    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:04.963577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.963516    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:04.963577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:04.963545    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:05.882120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:05.882081    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" event={"ID":"dca405bb-787c-44aa-83ae-cec31331ea1c","Type":"ContainerStarted","Data":"6988fe680d4b0c72953f4f9332ce85c07fc04ca5372042539f98ad51e5cfcdd3"}
Nov 02 01:42:05.882643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:05.882134    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" event={"ID":"dca405bb-787c-44aa-83ae-cec31331ea1c","Type":"ContainerStarted","Data":"4a4bfe3c4535e87f566a362bfddc74a182c7534faef0ee391bcbeeedffe042fb"}
Nov 02 01:42:05.910352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:05.910282    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" podStartSLOduration=133.910267583 podStartE2EDuration="2m13.910267583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:05.907417687 +0000 UTC m=+1631.212350523" watchObservedRunningTime="2026-11-02 01:42:05.910267583 +0000 UTC m=+1631.215200479"
Nov 02 01:42:06.608972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:06.608939    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:06.611162 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:06.611138    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 01:42:06.611320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:06.611199    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 01:42:06.872070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:06.871988    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb94e95-465c-49e8-9ad5-dc74fa3283bc" path="/var/lib/kubelet/pods/7cb94e95-465c-49e8-9ad5-dc74fa3283bc/volumes"
Nov 02 01:42:07.620317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:07.612526    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 01:42:07.620317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:07.612613    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 01:42:08.001316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:07.987676    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.068316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.067550    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.145318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.145185    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.173317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.173106    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.174521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.173557    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:42:08.517092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.517044    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.600456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.600403    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.614777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.614734    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 01:42:08.615049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.615022    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 01:42:08.713602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.713553    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.774021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.773906    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.833124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.833079    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.970312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.968927    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.989525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.988252    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:08.989525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:08.988787    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:42:09.070664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:09.070561    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:09.080327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:09.072410    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:42:09.098052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:09.098024    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:42:09.129495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:09.129464    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:42:09.609011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:09.608977    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 01:42:09.609197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:09.609041    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 01:42:10.609315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:10.608756    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 01:42:10.609315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:10.608827    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 01:42:11.608909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:11.608875    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 01:42:11.609100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:11.608929    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 01:42:12.609800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:12.609757    4851 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 01:42:12.610369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:12.609844    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 01:42:13.673440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:13.673365    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:13.674025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:13.673886    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:13.675720 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:13.675697    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:42:17.659820 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:42:17.659784    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:42:18.504790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.504758    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"]
Nov 02 01:42:18.510367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.510334    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:18.517567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.517537    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:18.517567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.517556    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:42:18.518762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.517584    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:42:18.518762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.517628    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:42:18.526601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.526574    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"]
Nov 02 01:42:18.577279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.577243    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2wl\" (UniqueName: \"kubernetes.io/projected/8c2d36c6-6b76-44df-9397-1d39991b3d07-kube-api-access-zh2wl\") pod \"maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:18.577279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.577311    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:18.577570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.577462    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:18.679330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.679282    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:18.679757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.679534    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2wl\" (UniqueName: \"kubernetes.io/projected/8c2d36c6-6b76-44df-9397-1d39991b3d07-kube-api-access-zh2wl\") pod \"maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:18.679757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.679564    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:18.680167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.680148    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:18.681764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.681743    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:18.752109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.752067    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2wl\" (UniqueName: \"kubernetes.io/projected/8c2d36c6-6b76-44df-9397-1d39991b3d07-kube-api-access-zh2wl\") pod \"maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:18.844989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:18.844910    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:19.134961 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:42:19.134929    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2d36c6_6b76_44df_9397_1d39991b3d07.slice/crio-0ca9d92d2ed35440aa3bf640c66b70ebb91625c6da32215b7a49bfdab0db1932 WatchSource:0}: Error finding container 0ca9d92d2ed35440aa3bf640c66b70ebb91625c6da32215b7a49bfdab0db1932: Status 404 returned error can't find the container with id 0ca9d92d2ed35440aa3bf640c66b70ebb91625c6da32215b7a49bfdab0db1932
Nov 02 01:42:19.136549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:19.136518    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"]
Nov 02 01:42:19.306481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:19.306447    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7" event={"ID":"8c2d36c6-6b76-44df-9397-1d39991b3d07","Type":"ContainerStarted","Data":"0ca9d92d2ed35440aa3bf640c66b70ebb91625c6da32215b7a49bfdab0db1932"}
Nov 02 01:42:20.317527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:20.317492    4851 generic.go:356] "Generic (PLEG): container finished" podID="8c2d36c6-6b76-44df-9397-1d39991b3d07" containerID="38d9605d30f59e6e8f48d4a74960a7fe765e27dfffda01ff2719f44cd478095e" exitCode=0
Nov 02 01:42:20.318005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:20.317606    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7" event={"ID":"8c2d36c6-6b76-44df-9397-1d39991b3d07","Type":"ContainerDied","Data":"38d9605d30f59e6e8f48d4a74960a7fe765e27dfffda01ff2719f44cd478095e"}
Nov 02 01:42:21.614419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:21.614395    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:21.674958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:21.674925    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-xks-ca\") pod \"8c2d36c6-6b76-44df-9397-1d39991b3d07\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") "
Nov 02 01:42:21.675158 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:21.674997    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2d36c6-6b76-44df-9397-1d39991b3d07-kube-api-access-zh2wl\" (UniqueName: \"kubernetes.io/projected/8c2d36c6-6b76-44df-9397-1d39991b3d07-kube-api-access-zh2wl\") pod \"8c2d36c6-6b76-44df-9397-1d39991b3d07\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") "
Nov 02 01:42:21.675158 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:21.675135    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-openshift-ca\") pod \"8c2d36c6-6b76-44df-9397-1d39991b3d07\" (UID: \"8c2d36c6-6b76-44df-9397-1d39991b3d07\") "
Nov 02 01:42:21.675751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:21.675718    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-openshift-ca" pod "8c2d36c6-6b76-44df-9397-1d39991b3d07" (UID: "8c2d36c6-6b76-44df-9397-1d39991b3d07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:21.675927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:21.675906    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:21.680311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:21.677921    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2d36c6-6b76-44df-9397-1d39991b3d07-kube-api-access-zh2wl" pod "8c2d36c6-6b76-44df-9397-1d39991b3d07" (UID: "8c2d36c6-6b76-44df-9397-1d39991b3d07"). InnerVolumeSpecName "kube-api-access-zh2wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:21.680311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:21.678368    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-xks-ca" pod "8c2d36c6-6b76-44df-9397-1d39991b3d07" (UID: "8c2d36c6-6b76-44df-9397-1d39991b3d07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:21.778169 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:21.778135    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2d36c6-6b76-44df-9397-1d39991b3d07-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:21.778169 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:21.778168    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh2wl\" (UniqueName: \"kubernetes.io/projected/8c2d36c6-6b76-44df-9397-1d39991b3d07-kube-api-access-zh2wl\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:22.376950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:22.376921    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"
Nov 02 01:42:22.376950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:22.376940    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7" event={"ID":"8c2d36c6-6b76-44df-9397-1d39991b3d07","Type":"ContainerDied","Data":"0ca9d92d2ed35440aa3bf640c66b70ebb91625c6da32215b7a49bfdab0db1932"}
Nov 02 01:42:22.377224 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:22.376983    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca9d92d2ed35440aa3bf640c66b70ebb91625c6da32215b7a49bfdab0db1932"
Nov 02 01:42:28.662072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:28.662022    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"]
Nov 02 01:42:29.944477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:29.944439    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"]
Nov 02 01:42:29.966892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:29.964680    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"]
Nov 02 01:42:33.019659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.019621    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"]
Nov 02 01:42:33.020629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.020610    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8c2d36c6-6b76-44df-9397-1d39991b3d07" containerName="revoke-keys"
Nov 02 01:42:33.020723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.020631    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2d36c6-6b76-44df-9397-1d39991b3d07" containerName="revoke-keys"
Nov 02 01:42:33.020812 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.020802    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2d36c6-6b76-44df-9397-1d39991b3d07" containerName="revoke-keys"
Nov 02 01:42:33.025107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.025090    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.029870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.029845    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:42:33.030028 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.029995    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:33.030525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.029954    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:42:33.030525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.029938    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:42:33.032486 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.032466    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"]
Nov 02 01:42:33.119773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.119733    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908a6786-06a1-4323-9157-3de0777f0627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1907de4a-2jxfw\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.119948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.119795    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908a6786-06a1-4323-9157-3de0777f0627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1907de4a-2jxfw\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.119995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.119965    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zf44\" (UniqueName: \"kubernetes.io/projected/908a6786-06a1-4323-9157-3de0777f0627-kube-api-access-5zf44\") pod \"maas-api-revoke-sub-1907de4a-2jxfw\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.221697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.221665    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908a6786-06a1-4323-9157-3de0777f0627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1907de4a-2jxfw\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.221893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.221721    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908a6786-06a1-4323-9157-3de0777f0627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1907de4a-2jxfw\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.221893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.221842    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zf44\" (UniqueName: \"kubernetes.io/projected/908a6786-06a1-4323-9157-3de0777f0627-kube-api-access-5zf44\") pod \"maas-api-revoke-sub-1907de4a-2jxfw\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.222416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.222395    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908a6786-06a1-4323-9157-3de0777f0627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1907de4a-2jxfw\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.224411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.224390    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908a6786-06a1-4323-9157-3de0777f0627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1907de4a-2jxfw\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.234146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.234116    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zf44\" (UniqueName: \"kubernetes.io/projected/908a6786-06a1-4323-9157-3de0777f0627-kube-api-access-5zf44\") pod \"maas-api-revoke-sub-1907de4a-2jxfw\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.341728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.341637    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:33.638466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.638432    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"]
Nov 02 01:42:33.662731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.662694    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8" podUID="d9c26f79-3936-4990-8b0a-3b7e2b32de0e" containerName="payload-processing" containerID="cri-o://fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732" gracePeriod=25
Nov 02 01:42:33.952466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:33.952439    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:42:34.075348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.075312    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-kube-api-access-clh9p\" (UniqueName: \"kubernetes.io/projected/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-kube-api-access-clh9p\") pod \"d9c26f79-3936-4990-8b0a-3b7e2b32de0e\" (UID: \"d9c26f79-3936-4990-8b0a-3b7e2b32de0e\") "
Nov 02 01:42:34.075700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.075479    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-plugins-config-volume\") pod \"d9c26f79-3936-4990-8b0a-3b7e2b32de0e\" (UID: \"d9c26f79-3936-4990-8b0a-3b7e2b32de0e\") "
Nov 02 01:42:34.076307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.076268    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-plugins-config-volume" pod "d9c26f79-3936-4990-8b0a-3b7e2b32de0e" (UID: "d9c26f79-3936-4990-8b0a-3b7e2b32de0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:34.078049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.078032    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-kube-api-access-clh9p" pod "d9c26f79-3936-4990-8b0a-3b7e2b32de0e" (UID: "d9c26f79-3936-4990-8b0a-3b7e2b32de0e"). InnerVolumeSpecName "kube-api-access-clh9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:34.177901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.177866    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:34.177901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.177899    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clh9p\" (UniqueName: \"kubernetes.io/projected/d9c26f79-3936-4990-8b0a-3b7e2b32de0e-kube-api-access-clh9p\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:34.198066 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.197993    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"]
Nov 02 01:42:34.198954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.198871    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d9c26f79-3936-4990-8b0a-3b7e2b32de0e" containerName="payload-processing"
Nov 02 01:42:34.198954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.198884    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c26f79-3936-4990-8b0a-3b7e2b32de0e" containerName="payload-processing"
Nov 02 01:42:34.199070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.199059    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c26f79-3936-4990-8b0a-3b7e2b32de0e" containerName="payload-processing"
Nov 02 01:42:34.203138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.203123    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.212337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.212247    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"]
Nov 02 01:42:34.279908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.279861    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c525c346-sh8k2\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.280085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.279969    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4cql\" (UniqueName: \"kubernetes.io/projected/74674aa0-450d-403d-8cde-95e91a71e7ab-kube-api-access-h4cql\") pod \"maas-api-revoke-sub-c525c346-sh8k2\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.280229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.280103    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c525c346-sh8k2\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.395406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.390197    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c525c346-sh8k2\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.395406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.390271    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cql\" (UniqueName: \"kubernetes.io/projected/74674aa0-450d-403d-8cde-95e91a71e7ab-kube-api-access-h4cql\") pod \"maas-api-revoke-sub-c525c346-sh8k2\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.395406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.390394    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c525c346-sh8k2\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.396987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.396011    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c525c346-sh8k2\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.400596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.400537    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c525c346-sh8k2\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.440312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.439841    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4cql\" (UniqueName: \"kubernetes.io/projected/74674aa0-450d-403d-8cde-95e91a71e7ab-kube-api-access-h4cql\") pod \"maas-api-revoke-sub-c525c346-sh8k2\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.528344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.528245    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:34.595123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.594831    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw" event={"ID":"908a6786-06a1-4323-9157-3de0777f0627","Type":"ContainerStarted","Data":"d5b90d616c4649cbe8af42870838afb6b74cf796827bb2341a34cc75a94255ee"}
Nov 02 01:42:34.595123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.594896    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw" event={"ID":"908a6786-06a1-4323-9157-3de0777f0627","Type":"ContainerStarted","Data":"954db34a0c93fe64c38f321ea3d1230209f2b82ca32bfc9132a9674f0fe3a4cc"}
Nov 02 01:42:34.596765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.596739    4851 generic.go:356] "Generic (PLEG): container finished" podID="d9c26f79-3936-4990-8b0a-3b7e2b32de0e" containerID="fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732" exitCode=0
Nov 02 01:42:34.596883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.596767    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"
Nov 02 01:42:34.596883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.596794    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8" event={"ID":"d9c26f79-3936-4990-8b0a-3b7e2b32de0e","Type":"ContainerDied","Data":"fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732"}
Nov 02 01:42:34.596883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.596841    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8" event={"ID":"d9c26f79-3936-4990-8b0a-3b7e2b32de0e","Type":"ContainerDied","Data":"0174dc1549c1d1453a249e028baa2b5784debdca8a800f3f6c57747e9311d6b0"}
Nov 02 01:42:34.596883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.596864    4851 scope.go:122] "RemoveContainer" containerID="fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732"
Nov 02 01:42:34.620659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.620627    4851 scope.go:122] "RemoveContainer" containerID="fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732"
Nov 02 01:42:34.626184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.626131    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw" podStartSLOduration=2.626114382 podStartE2EDuration="2.626114382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:34.622633227 +0000 UTC m=+1659.927566054" watchObservedRunningTime="2026-11-02 01:42:34.626114382 +0000 UTC m=+1659.931047209"
Nov 02 01:42:34.626890 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:42:34.626860    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732\": container with ID starting with fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732 not found: ID does not exist" containerID="fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732"
Nov 02 01:42:34.626990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.626902    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732"} err="failed to get container status \"fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732\": rpc error: code = NotFound desc = could not find container \"fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732\": container with ID starting with fa98eb786c3cf7963ee1fc5ed0fd04e7d128b49f73cb5cf308776f131cbd6732 not found: ID does not exist"
Nov 02 01:42:34.648370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.648342    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"]
Nov 02 01:42:34.655109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.655083    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edef28-766f5fb765-xbnw8"]
Nov 02 01:42:34.817211 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:42:34.816815    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74674aa0_450d_403d_8cde_95e91a71e7ab.slice/crio-6545de25ac0acfa2a673ef114b3d1ddfbad0dc644e38c39d638e29f15fd92019 WatchSource:0}: Error finding container 6545de25ac0acfa2a673ef114b3d1ddfbad0dc644e38c39d638e29f15fd92019: Status 404 returned error can't find the container with id 6545de25ac0acfa2a673ef114b3d1ddfbad0dc644e38c39d638e29f15fd92019
Nov 02 01:42:34.817211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.816952    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"]
Nov 02 01:42:34.850516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.850493    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c26f79-3936-4990-8b0a-3b7e2b32de0e" path="/var/lib/kubelet/pods/d9c26f79-3936-4990-8b0a-3b7e2b32de0e/volumes"
Nov 02 01:42:34.945115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.945076    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt" podUID="5be589a6-1443-4701-a4e6-0a5789a69eec" containerName="payload-processing" containerID="cri-o://6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a" gracePeriod=25
Nov 02 01:42:34.965547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:34.965513    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k" podUID="01c06bfd-cd8f-49ef-ad57-fc5653a65e7b" containerName="payload-pre-processing" containerID="cri-o://468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0" gracePeriod=25
Nov 02 01:42:35.386519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.386492    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:42:35.478273 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.478238    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:42:35.526055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.526021    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be589a6-1443-4701-a4e6-0a5789a69eec-kube-api-access-4scpq\" (UniqueName: \"kubernetes.io/projected/5be589a6-1443-4701-a4e6-0a5789a69eec-kube-api-access-4scpq\") pod \"5be589a6-1443-4701-a4e6-0a5789a69eec\" (UID: \"5be589a6-1443-4701-a4e6-0a5789a69eec\") "
Nov 02 01:42:35.526249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.526230    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be589a6-1443-4701-a4e6-0a5789a69eec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be589a6-1443-4701-a4e6-0a5789a69eec-plugins-config-volume\") pod \"5be589a6-1443-4701-a4e6-0a5789a69eec\" (UID: \"5be589a6-1443-4701-a4e6-0a5789a69eec\") "
Nov 02 01:42:35.531756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.531396    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"]
Nov 02 01:42:35.532344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.532259    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be589a6-1443-4701-a4e6-0a5789a69eec-plugins-config-volume" pod "5be589a6-1443-4701-a4e6-0a5789a69eec" (UID: "5be589a6-1443-4701-a4e6-0a5789a69eec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:35.532680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.532659    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5be589a6-1443-4701-a4e6-0a5789a69eec" containerName="payload-processing"
Nov 02 01:42:35.532780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.532684    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be589a6-1443-4701-a4e6-0a5789a69eec" containerName="payload-processing"
Nov 02 01:42:35.532780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.532741    4851 cpu_manager.go:414] "containerMap: removing container" podUID="01c06bfd-cd8f-49ef-ad57-fc5653a65e7b" containerName="payload-pre-processing"
Nov 02 01:42:35.532780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.532750    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c06bfd-cd8f-49ef-ad57-fc5653a65e7b" containerName="payload-pre-processing"
Nov 02 01:42:35.533019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.533005    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c06bfd-cd8f-49ef-ad57-fc5653a65e7b" containerName="payload-pre-processing"
Nov 02 01:42:35.533084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.533032    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be589a6-1443-4701-a4e6-0a5789a69eec" containerName="payload-processing"
Nov 02 01:42:35.537679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.537651    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be589a6-1443-4701-a4e6-0a5789a69eec-kube-api-access-4scpq" pod "5be589a6-1443-4701-a4e6-0a5789a69eec" (UID: "5be589a6-1443-4701-a4e6-0a5789a69eec"). InnerVolumeSpecName "kube-api-access-4scpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:35.538225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.538207    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"]
Nov 02 01:42:35.538348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.538320    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.610829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.610748    4851 generic.go:356] "Generic (PLEG): container finished" podID="01c06bfd-cd8f-49ef-ad57-fc5653a65e7b" containerID="468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0" exitCode=0
Nov 02 01:42:35.611000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.610824    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"
Nov 02 01:42:35.611000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.610841    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k" event={"ID":"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b","Type":"ContainerDied","Data":"468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0"}
Nov 02 01:42:35.611000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.610907    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k" event={"ID":"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b","Type":"ContainerDied","Data":"85b9cffbd6be7649ee62634ef349dc00749423cd7a974aac89cb286f7ab31fbc"}
Nov 02 01:42:35.611372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.610928    4851 scope.go:122] "RemoveContainer" containerID="468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0"
Nov 02 01:42:35.612834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.612595    4851 generic.go:356] "Generic (PLEG): container finished" podID="5be589a6-1443-4701-a4e6-0a5789a69eec" containerID="6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a" exitCode=0
Nov 02 01:42:35.612834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.612652    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"
Nov 02 01:42:35.613100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.613077    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt" event={"ID":"5be589a6-1443-4701-a4e6-0a5789a69eec","Type":"ContainerDied","Data":"6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a"}
Nov 02 01:42:35.613181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.613128    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt" event={"ID":"5be589a6-1443-4701-a4e6-0a5789a69eec","Type":"ContainerDied","Data":"419e7e772a90b2708dfdbfc34f737d49e49fe35359e2bba8b4cf022f68719c2f"}
Nov 02 01:42:35.615534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.615507    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2" event={"ID":"74674aa0-450d-403d-8cde-95e91a71e7ab","Type":"ContainerStarted","Data":"926cd98d84691d438a7392fc09ad6cf5a68b6b1a43995a27b38e38384dd65b3a"}
Nov 02 01:42:35.615657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.615555    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2" event={"ID":"74674aa0-450d-403d-8cde-95e91a71e7ab","Type":"ContainerStarted","Data":"6545de25ac0acfa2a673ef114b3d1ddfbad0dc644e38c39d638e29f15fd92019"}
Nov 02 01:42:35.617528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.617448    4851 generic.go:356] "Generic (PLEG): container finished" podID="908a6786-06a1-4323-9157-3de0777f0627" containerID="d5b90d616c4649cbe8af42870838afb6b74cf796827bb2341a34cc75a94255ee" exitCode=0
Nov 02 01:42:35.617657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.617627    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw" event={"ID":"908a6786-06a1-4323-9157-3de0777f0627","Type":"ContainerDied","Data":"d5b90d616c4649cbe8af42870838afb6b74cf796827bb2341a34cc75a94255ee"}
Nov 02 01:42:35.625792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.625776    4851 scope.go:122] "RemoveContainer" containerID="468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0"
Nov 02 01:42:35.626065 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:42:35.626048    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0\": container with ID starting with 468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0 not found: ID does not exist" containerID="468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0"
Nov 02 01:42:35.626112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.626070    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0"} err="failed to get container status \"468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0\": rpc error: code = NotFound desc = could not find container \"468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0\": container with ID starting with 468b77686d43610fea891a0d212ef12a13a15fd4cf5fd77e1966fde75a7b0bd0 not found: ID does not exist"
Nov 02 01:42:35.626112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.626086    4851 scope.go:122] "RemoveContainer" containerID="6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a"
Nov 02 01:42:35.632802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.632783    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-plugins-config-volume\") pod \"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b\" (UID: \"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b\") "
Nov 02 01:42:35.632924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.632828    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-kube-api-access-jlsl6\" (UniqueName: \"kubernetes.io/projected/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-kube-api-access-jlsl6\") pod \"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b\" (UID: \"01c06bfd-cd8f-49ef-ad57-fc5653a65e7b\") "
Nov 02 01:42:35.633167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.633145    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-plugins-config-volume" pod "01c06bfd-cd8f-49ef-ad57-fc5653a65e7b" (UID: "01c06bfd-cd8f-49ef-ad57-fc5653a65e7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:35.633263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.633243    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.633423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.633408    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.633503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.633440    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.633589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.633576    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.633651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.633630    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.633713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.633701    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqvd5\" (UniqueName: \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-kube-api-access-nqvd5\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.633820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.633803    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.633856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.633834    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.633973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.633957    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.634154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.634138    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be589a6-1443-4701-a4e6-0a5789a69eec-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:35.634221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.634161    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:35.634221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.634177    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4scpq\" (UniqueName: \"kubernetes.io/projected/5be589a6-1443-4701-a4e6-0a5789a69eec-kube-api-access-4scpq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:35.635051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.635030    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-kube-api-access-jlsl6" pod "01c06bfd-cd8f-49ef-ad57-fc5653a65e7b" (UID: "01c06bfd-cd8f-49ef-ad57-fc5653a65e7b"). InnerVolumeSpecName "kube-api-access-jlsl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:35.639603 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.639585    4851 scope.go:122] "RemoveContainer" containerID="6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a"
Nov 02 01:42:35.639902 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:42:35.639885    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a\": container with ID starting with 6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a not found: ID does not exist" containerID="6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a"
Nov 02 01:42:35.639943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.639909    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a"} err="failed to get container status \"6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a\": rpc error: code = NotFound desc = could not find container \"6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a\": container with ID starting with 6a033aaff77fd2458b3eb09aa94dacd6ffb58fd619788c66d73654ba2e63e30a not found: ID does not exist"
Nov 02 01:42:35.676596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.676543    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2" podStartSLOduration=1.676523921 podStartE2EDuration="1.676523921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:35.661216791 +0000 UTC m=+1660.966149619" watchObservedRunningTime="2026-11-02 01:42:35.676523921 +0000 UTC m=+1660.981456748"
Nov 02 01:42:35.680963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.680935    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"]
Nov 02 01:42:35.685735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.685709    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8b3bae-8bb786b59-h7ckt"]
Nov 02 01:42:35.772872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.742747    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.772872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.770978    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqvd5\" (UniqueName: \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-kube-api-access-nqvd5\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.772872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.771107    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.772872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.771144    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.772872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.771270    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.785371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.785118    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.803476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.795621    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.803476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.796038    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.816355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.815707    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.816355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.815882    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.816355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.815916    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.816355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.816023    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.816355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.816185    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlsl6\" (UniqueName: \"kubernetes.io/projected/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b-kube-api-access-jlsl6\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:35.816799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.816513    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.842315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.817257    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.842595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.822057    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.842749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.824645    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqvd5\" (UniqueName: \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-kube-api-access-nqvd5\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.854089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.854023    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:35.865770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:35.865696    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:36.158466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.158366    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"]
Nov 02 01:42:36.184492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.184399    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:36.204326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.196220    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8b3bae-78b4dd9595-d9h4k"]
Nov 02 01:42:36.389972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.389926    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"]
Nov 02 01:42:36.403168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.403136    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-edef28-s2qr7"]
Nov 02 01:42:36.646043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.645089    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"]
Nov 02 01:42:36.684809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.684767    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:36.686077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.684883    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:36.686077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.684923    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:36.686077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.685403    4851 generic.go:356] "Generic (PLEG): container finished" podID="74674aa0-450d-403d-8cde-95e91a71e7ab" containerID="926cd98d84691d438a7392fc09ad6cf5a68b6b1a43995a27b38e38384dd65b3a" exitCode=0
Nov 02 01:42:36.686077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.685643    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2" event={"ID":"74674aa0-450d-403d-8cde-95e91a71e7ab","Type":"ContainerDied","Data":"926cd98d84691d438a7392fc09ad6cf5a68b6b1a43995a27b38e38384dd65b3a"}
Nov 02 01:42:36.874491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.874459    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c06bfd-cd8f-49ef-ad57-fc5653a65e7b" path="/var/lib/kubelet/pods/01c06bfd-cd8f-49ef-ad57-fc5653a65e7b/volumes"
Nov 02 01:42:36.875440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.875411    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be589a6-1443-4701-a4e6-0a5789a69eec" path="/var/lib/kubelet/pods/5be589a6-1443-4701-a4e6-0a5789a69eec/volumes"
Nov 02 01:42:36.876263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.876245    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2d36c6-6b76-44df-9397-1d39991b3d07" path="/var/lib/kubelet/pods/8c2d36c6-6b76-44df-9397-1d39991b3d07/volumes"
Nov 02 01:42:36.961389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:36.961340    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:37.107912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.107877    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908a6786-06a1-4323-9157-3de0777f0627-kube-api-access-5zf44\" (UniqueName: \"kubernetes.io/projected/908a6786-06a1-4323-9157-3de0777f0627-kube-api-access-5zf44\") pod \"908a6786-06a1-4323-9157-3de0777f0627\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") "
Nov 02 01:42:37.108120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.108104    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908a6786-06a1-4323-9157-3de0777f0627-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908a6786-06a1-4323-9157-3de0777f0627-maas-api-openshift-ca\") pod \"908a6786-06a1-4323-9157-3de0777f0627\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") "
Nov 02 01:42:37.108186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.108147    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908a6786-06a1-4323-9157-3de0777f0627-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908a6786-06a1-4323-9157-3de0777f0627-maas-api-xks-ca\") pod \"908a6786-06a1-4323-9157-3de0777f0627\" (UID: \"908a6786-06a1-4323-9157-3de0777f0627\") "
Nov 02 01:42:37.108548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.108517    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908a6786-06a1-4323-9157-3de0777f0627-maas-api-openshift-ca" pod "908a6786-06a1-4323-9157-3de0777f0627" (UID: "908a6786-06a1-4323-9157-3de0777f0627"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:37.109115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.109095    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908a6786-06a1-4323-9157-3de0777f0627-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:37.110475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.110452    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908a6786-06a1-4323-9157-3de0777f0627-kube-api-access-5zf44" pod "908a6786-06a1-4323-9157-3de0777f0627" (UID: "908a6786-06a1-4323-9157-3de0777f0627"). InnerVolumeSpecName "kube-api-access-5zf44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:37.111004 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.110986    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908a6786-06a1-4323-9157-3de0777f0627-maas-api-xks-ca" pod "908a6786-06a1-4323-9157-3de0777f0627" (UID: "908a6786-06a1-4323-9157-3de0777f0627"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:37.222199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.222110    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zf44\" (UniqueName: \"kubernetes.io/projected/908a6786-06a1-4323-9157-3de0777f0627-kube-api-access-5zf44\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:37.222442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.222422    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908a6786-06a1-4323-9157-3de0777f0627-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:37.902368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.902334    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw" event={"ID":"908a6786-06a1-4323-9157-3de0777f0627","Type":"ContainerDied","Data":"954db34a0c93fe64c38f321ea3d1230209f2b82ca32bfc9132a9674f0fe3a4cc"}
Nov 02 01:42:37.903813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.903794    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="954db34a0c93fe64c38f321ea3d1230209f2b82ca32bfc9132a9674f0fe3a4cc"
Nov 02 01:42:37.904082 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.902470    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"
Nov 02 01:42:37.908428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.908328    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" event={"ID":"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e","Type":"ContainerStarted","Data":"616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5"}
Nov 02 01:42:37.908428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:37.908379    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" event={"ID":"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e","Type":"ContainerStarted","Data":"3ed15a638d0a1d0390841d712bfb15095ad887c43e041370d4c454ad371a1e1c"}
Nov 02 01:42:38.186138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.184988    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:38.188100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.188066    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 01:42:38.188261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.188121    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 01:42:38.330678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.330637    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:38.364327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.359865    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podStartSLOduration=165.359844614 podStartE2EDuration="2m45.359844614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:37.951581121 +0000 UTC m=+1663.256513948" watchObservedRunningTime="2026-11-02 01:42:38.359844614 +0000 UTC m=+1663.664777442"
Nov 02 01:42:38.410144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.410104    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-openshift-ca\") pod \"74674aa0-450d-403d-8cde-95e91a71e7ab\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") "
Nov 02 01:42:38.410455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.410436    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74674aa0-450d-403d-8cde-95e91a71e7ab-kube-api-access-h4cql\" (UniqueName: \"kubernetes.io/projected/74674aa0-450d-403d-8cde-95e91a71e7ab-kube-api-access-h4cql\") pod \"74674aa0-450d-403d-8cde-95e91a71e7ab\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") "
Nov 02 01:42:38.410699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.410683    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-xks-ca\") pod \"74674aa0-450d-403d-8cde-95e91a71e7ab\" (UID: \"74674aa0-450d-403d-8cde-95e91a71e7ab\") "
Nov 02 01:42:38.412877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.412845    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-openshift-ca" pod "74674aa0-450d-403d-8cde-95e91a71e7ab" (UID: "74674aa0-450d-403d-8cde-95e91a71e7ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:38.464230 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.464190    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74674aa0-450d-403d-8cde-95e91a71e7ab-kube-api-access-h4cql" pod "74674aa0-450d-403d-8cde-95e91a71e7ab" (UID: "74674aa0-450d-403d-8cde-95e91a71e7ab"). InnerVolumeSpecName "kube-api-access-h4cql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:38.495713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.495678    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-xks-ca" pod "74674aa0-450d-403d-8cde-95e91a71e7ab" (UID: "74674aa0-450d-403d-8cde-95e91a71e7ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:38.615318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.589939    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:38.615318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.598416    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74674aa0-450d-403d-8cde-95e91a71e7ab-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:38.615318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.598436    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4cql\" (UniqueName: \"kubernetes.io/projected/74674aa0-450d-403d-8cde-95e91a71e7ab-kube-api-access-h4cql\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:38.917237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.917208    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"
Nov 02 01:42:38.917667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.917232    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2" event={"ID":"74674aa0-450d-403d-8cde-95e91a71e7ab","Type":"ContainerDied","Data":"6545de25ac0acfa2a673ef114b3d1ddfbad0dc644e38c39d638e29f15fd92019"}
Nov 02 01:42:38.917667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:38.917280    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6545de25ac0acfa2a673ef114b3d1ddfbad0dc644e38c39d638e29f15fd92019"
Nov 02 01:42:39.192314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:39.192118    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 01:42:39.192314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:39.192182    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 01:42:40.185458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:40.185426    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 01:42:40.185861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:40.185485    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 01:42:41.185215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:41.185185    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 01:42:41.185395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:41.185238    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 01:42:42.185884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:42.185132    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 01:42:42.185884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:42.185188    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 01:42:43.185321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:43.185076    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 01:42:43.185321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:43.185123    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 01:42:44.120314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.111521    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"]
Nov 02 01:42:44.121320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.121257    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" containerID="cri-o://e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e" gracePeriod=30
Nov 02 01:42:44.264316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.223026    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 01:42:44.264316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.224539    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 01:42:44.444726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.444679    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"]
Nov 02 01:42:44.702833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.702744    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"]
Nov 02 01:42:44.729322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.704456    4851 cpu_manager.go:414] "containerMap: removing container" podUID="908a6786-06a1-4323-9157-3de0777f0627" containerName="revoke-keys"
Nov 02 01:42:44.730058 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.730024    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908a6786-06a1-4323-9157-3de0777f0627" containerName="revoke-keys"
Nov 02 01:42:44.730236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.730223    4851 cpu_manager.go:414] "containerMap: removing container" podUID="74674aa0-450d-403d-8cde-95e91a71e7ab" containerName="revoke-keys"
Nov 02 01:42:44.730359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.730345    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74674aa0-450d-403d-8cde-95e91a71e7ab" containerName="revoke-keys"
Nov 02 01:42:44.730835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.730815    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="908a6786-06a1-4323-9157-3de0777f0627" containerName="revoke-keys"
Nov 02 01:42:44.730964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.730953    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="74674aa0-450d-403d-8cde-95e91a71e7ab" containerName="revoke-keys"
Nov 02 01:42:44.926379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.923423    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:42:44.926379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.924734    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"]
Nov 02 01:42:44.960159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:44.959428    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:42:45.045612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.033179    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vtc\" (UniqueName: \"kubernetes.io/projected/bb6460e6-9466-45fe-a13c-22a1ae10e912-kube-api-access-54vtc\") pod \"maas-api-revoke-sub-54f57396-vwfn5\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.046111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.046081    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54f57396-vwfn5\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.046255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.046191    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54f57396-vwfn5\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.091138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.091107    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.115869 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.115839    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:42:45.116187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.116171    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:45.116399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.116365    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:45.117178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.116384    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:42:45.137729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.136894    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:45.164354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.163743    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vtc\" (UniqueName: \"kubernetes.io/projected/bb6460e6-9466-45fe-a13c-22a1ae10e912-kube-api-access-54vtc\") pod \"maas-api-revoke-sub-54f57396-vwfn5\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.165629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.165608    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54f57396-vwfn5\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.165877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.165845    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54f57396-vwfn5\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.167193 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.167078    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54f57396-vwfn5\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.171255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.169608    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:45.171255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.170803    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:42:45.222012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.210593    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 01:42:45.222012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.210653    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 01:42:45.222012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.213660    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:42:45.222012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.221932    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54f57396-vwfn5\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.230986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.230954    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vtc\" (UniqueName: \"kubernetes.io/projected/bb6460e6-9466-45fe-a13c-22a1ae10e912-kube-api-access-54vtc\") pod \"maas-api-revoke-sub-54f57396-vwfn5\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.439802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.439765    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"]
Nov 02 01:42:45.446206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.446152    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.454107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.454074    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"]
Nov 02 01:42:45.487391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.487234    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f47a381-6kljw\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.487524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.487399    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxk9\" (UniqueName: \"kubernetes.io/projected/6650af1a-9d41-48df-9df0-dafa1d85a2cc-kube-api-access-5rxk9\") pod \"maas-api-revoke-sub-0f47a381-6kljw\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.487735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.487715    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f47a381-6kljw\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.520220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.520186    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:45.589956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.589908    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f47a381-6kljw\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.590937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.590367    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxk9\" (UniqueName: \"kubernetes.io/projected/6650af1a-9d41-48df-9df0-dafa1d85a2cc-kube-api-access-5rxk9\") pod \"maas-api-revoke-sub-0f47a381-6kljw\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.590937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.590751    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f47a381-6kljw\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.592190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.592016    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f47a381-6kljw\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.595550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.594308    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f47a381-6kljw\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.605932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.604968    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxk9\" (UniqueName: \"kubernetes.io/projected/6650af1a-9d41-48df-9df0-dafa1d85a2cc-kube-api-access-5rxk9\") pod \"maas-api-revoke-sub-0f47a381-6kljw\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.766898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.766812    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:45.825705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:45.825675    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"]
Nov 02 01:42:45.826671 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:42:45.826636    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6460e6_9466_45fe_a13c_22a1ae10e912.slice/crio-4f45c25540beafe30f09e9431ff24074f344b5f1ee3b0ed89e8f6a5f18ace4ec WatchSource:0}: Error finding container 4f45c25540beafe30f09e9431ff24074f344b5f1ee3b0ed89e8f6a5f18ace4ec: Status 404 returned error can't find the container with id 4f45c25540beafe30f09e9431ff24074f344b5f1ee3b0ed89e8f6a5f18ace4ec
Nov 02 01:42:46.217042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:46.201838    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:46.217042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:46.215344    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:46.223239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:46.223206    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:42:46.245530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:46.242466    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"]
Nov 02 01:42:46.497482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:46.496749    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw" event={"ID":"6650af1a-9d41-48df-9df0-dafa1d85a2cc","Type":"ContainerStarted","Data":"54b9339af7749998c60b82d6826988b1d5637aadffd285dbf2d968e03c54867b"}
Nov 02 01:42:46.500843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:46.500815    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5" event={"ID":"bb6460e6-9466-45fe-a13c-22a1ae10e912","Type":"ContainerStarted","Data":"4f45c25540beafe30f09e9431ff24074f344b5f1ee3b0ed89e8f6a5f18ace4ec"}
Nov 02 01:42:47.511611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:47.511408    4851 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:42:47.511611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:47.511473    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:42:47.547378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:47.547340    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw" event={"ID":"6650af1a-9d41-48df-9df0-dafa1d85a2cc","Type":"ContainerStarted","Data":"c6d50170ffa54982ee21e9736917e435d663782addc69b7c7de5524fc53195b2"}
Nov 02 01:42:47.549501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:47.549474    4851 generic.go:356] "Generic (PLEG): container finished" podID="bb6460e6-9466-45fe-a13c-22a1ae10e912" containerID="cdc536eea8190e9f6782349bc52219be0bd34716485681cc61227df11bb40fb0" exitCode=0
Nov 02 01:42:47.549636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:47.549559    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5" event={"ID":"bb6460e6-9466-45fe-a13c-22a1ae10e912","Type":"ContainerDied","Data":"cdc536eea8190e9f6782349bc52219be0bd34716485681cc61227df11bb40fb0"}
Nov 02 01:42:47.571925 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:47.571880    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw" podStartSLOduration=2.571865839 podStartE2EDuration="2.571865839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:47.565435771 +0000 UTC m=+1672.870368597" watchObservedRunningTime="2026-11-02 01:42:47.571865839 +0000 UTC m=+1672.876798665"
Nov 02 01:42:47.666243 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:42:47.666205    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:42:48.560370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:48.560329    4851 generic.go:356] "Generic (PLEG): container finished" podID="6650af1a-9d41-48df-9df0-dafa1d85a2cc" containerID="c6d50170ffa54982ee21e9736917e435d663782addc69b7c7de5524fc53195b2" exitCode=0
Nov 02 01:42:48.560833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:48.560410    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw" event={"ID":"6650af1a-9d41-48df-9df0-dafa1d85a2cc","Type":"ContainerDied","Data":"c6d50170ffa54982ee21e9736917e435d663782addc69b7c7de5524fc53195b2"}
Nov 02 01:42:48.777686 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:48.777660    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:48.893143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:48.893074    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-openshift-ca\") pod \"bb6460e6-9466-45fe-a13c-22a1ae10e912\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") "
Nov 02 01:42:48.893318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:48.893222    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6460e6-9466-45fe-a13c-22a1ae10e912-kube-api-access-54vtc\" (UniqueName: \"kubernetes.io/projected/bb6460e6-9466-45fe-a13c-22a1ae10e912-kube-api-access-54vtc\") pod \"bb6460e6-9466-45fe-a13c-22a1ae10e912\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") "
Nov 02 01:42:48.893318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:48.893304    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-xks-ca\") pod \"bb6460e6-9466-45fe-a13c-22a1ae10e912\" (UID: \"bb6460e6-9466-45fe-a13c-22a1ae10e912\") "
Nov 02 01:42:48.893514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:48.893493    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-openshift-ca" pod "bb6460e6-9466-45fe-a13c-22a1ae10e912" (UID: "bb6460e6-9466-45fe-a13c-22a1ae10e912"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:48.894420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:48.894394    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:48.895631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:48.895602    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-xks-ca" pod "bb6460e6-9466-45fe-a13c-22a1ae10e912" (UID: "bb6460e6-9466-45fe-a13c-22a1ae10e912"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:48.895764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:48.895651    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6460e6-9466-45fe-a13c-22a1ae10e912-kube-api-access-54vtc" pod "bb6460e6-9466-45fe-a13c-22a1ae10e912" (UID: "bb6460e6-9466-45fe-a13c-22a1ae10e912"). InnerVolumeSpecName "kube-api-access-54vtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:49.007943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.005507    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vtc\" (UniqueName: \"kubernetes.io/projected/bb6460e6-9466-45fe-a13c-22a1ae10e912-kube-api-access-54vtc\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.007943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.005562    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6460e6-9466-45fe-a13c-22a1ae10e912-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.517987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.517956    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:42:49.556490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.556412    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"]
Nov 02 01:42:49.558567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.558541    4851 cpu_manager.go:414] "containerMap: removing container" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy"
Nov 02 01:42:49.558567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.558566    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy"
Nov 02 01:42:49.558801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.558637    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bb6460e6-9466-45fe-a13c-22a1ae10e912" containerName="revoke-keys"
Nov 02 01:42:49.558801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.558645    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6460e6-9466-45fe-a13c-22a1ae10e912" containerName="revoke-keys"
Nov 02 01:42:49.558909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.558824    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6460e6-9466-45fe-a13c-22a1ae10e912" containerName="revoke-keys"
Nov 02 01:42:49.558909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.558839    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy"
Nov 02 01:42:49.558909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.558848    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy"
Nov 02 01:42:49.564914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.564683    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.573045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.572861    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8qwrd\""
Nov 02 01:42:49.573045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.572900    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-14b63f-serving-cert\""
Nov 02 01:42:49.573045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.573008    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 01:42:49.586956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.586929    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"]
Nov 02 01:42:49.594968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.594944    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"
Nov 02 01:42:49.595100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.594970    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5" event={"ID":"bb6460e6-9466-45fe-a13c-22a1ae10e912","Type":"ContainerDied","Data":"4f45c25540beafe30f09e9431ff24074f344b5f1ee3b0ed89e8f6a5f18ace4ec"}
Nov 02 01:42:49.595100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.595022    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f45c25540beafe30f09e9431ff24074f344b5f1ee3b0ed89e8f6a5f18ace4ec"
Nov 02 01:42:49.597403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.597378    4851 generic.go:356] "Generic (PLEG): container finished" podID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerID="e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e" exitCode=0
Nov 02 01:42:49.597529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.597513    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"
Nov 02 01:42:49.597619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.597579    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" event={"ID":"548143a8-0c95-41f1-9c60-857747c3c4dd","Type":"ContainerDied","Data":"e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e"}
Nov 02 01:42:49.597687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.597629    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z" event={"ID":"548143a8-0c95-41f1-9c60-857747c3c4dd","Type":"ContainerDied","Data":"910adbc47490f2b921cbd316dfce5482881460b445363522a0cdcb5c31736ea1"}
Nov 02 01:42:49.597687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.597661    4851 scope.go:122] "RemoveContainer" containerID="e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e"
Nov 02 01:42:49.614063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.614039    4851 scope.go:122] "RemoveContainer" containerID="f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85"
Nov 02 01:42:49.615495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.615468    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-certs\") pod \"548143a8-0c95-41f1-9c60-857747c3c4dd\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") "
Nov 02 01:42:49.615653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.615634    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-data\") pod \"548143a8-0c95-41f1-9c60-857747c3c4dd\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") "
Nov 02 01:42:49.615943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.615671    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-kube-api-access-vqklt\" (UniqueName: \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-kube-api-access-vqklt\") pod \"548143a8-0c95-41f1-9c60-857747c3c4dd\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") "
Nov 02 01:42:49.615943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.615722    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-podinfo\") pod \"548143a8-0c95-41f1-9c60-857747c3c4dd\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") "
Nov 02 01:42:49.615943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.615775    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-socket\") pod \"548143a8-0c95-41f1-9c60-857747c3c4dd\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") "
Nov 02 01:42:49.615943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.615844    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-token\" (UniqueName: \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-token\") pod \"548143a8-0c95-41f1-9c60-857747c3c4dd\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") "
Nov 02 01:42:49.616266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.615962    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-credential-socket\") pod \"548143a8-0c95-41f1-9c60-857747c3c4dd\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") "
Nov 02 01:42:49.616266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.616158    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-envoy\") pod \"548143a8-0c95-41f1-9c60-857747c3c4dd\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") "
Nov 02 01:42:49.616266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.616201    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548143a8-0c95-41f1-9c60-857747c3c4dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548143a8-0c95-41f1-9c60-857747c3c4dd-istiod-ca-cert\") pod \"548143a8-0c95-41f1-9c60-857747c3c4dd\" (UID: \"548143a8-0c95-41f1-9c60-857747c3c4dd\") "
Nov 02 01:42:49.618445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.617237    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-socket" pod "548143a8-0c95-41f1-9c60-857747c3c4dd" (UID: "548143a8-0c95-41f1-9c60-857747c3c4dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:49.618445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.617447    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-credential-socket" pod "548143a8-0c95-41f1-9c60-857747c3c4dd" (UID: "548143a8-0c95-41f1-9c60-857747c3c4dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:49.618445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.617472    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9sp\" (UniqueName: \"kubernetes.io/projected/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-kube-api-access-dd9sp\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.618445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.617603    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-maas-api-tls\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.618445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.617844    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.618445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.618092    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-metrics-certs\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.618445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.618219    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.618445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.618239    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.618445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.618259    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548143a8-0c95-41f1-9c60-857747c3c4dd-istiod-ca-cert" pod "548143a8-0c95-41f1-9c60-857747c3c4dd" (UID: "548143a8-0c95-41f1-9c60-857747c3c4dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:49.619587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.618560    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-data" pod "548143a8-0c95-41f1-9c60-857747c3c4dd" (UID: "548143a8-0c95-41f1-9c60-857747c3c4dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:49.619587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.618650    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-certs" pod "548143a8-0c95-41f1-9c60-857747c3c4dd" (UID: "548143a8-0c95-41f1-9c60-857747c3c4dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:49.621271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.621246    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-token" pod "548143a8-0c95-41f1-9c60-857747c3c4dd" (UID: "548143a8-0c95-41f1-9c60-857747c3c4dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:49.631493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.631405    4851 scope.go:122] "RemoveContainer" containerID="e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e"
Nov 02 01:42:49.632042 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:42:49.631917    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e\": container with ID starting with e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e not found: ID does not exist" containerID="e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e"
Nov 02 01:42:49.632042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.631951    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e"} err="failed to get container status \"e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e\": rpc error: code = NotFound desc = could not find container \"e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e\": container with ID starting with e88cc939a79f3106dc29e852463b5c03f18136d1cbc25901b94bcef6e102973e not found: ID does not exist"
Nov 02 01:42:49.632042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.631975    4851 scope.go:122] "RemoveContainer" containerID="f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85"
Nov 02 01:42:49.632435 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:42:49.632373    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85\": container with ID starting with f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85 not found: ID does not exist" containerID="f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85"
Nov 02 01:42:49.632435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.632401    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85"} err="failed to get container status \"f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85\": rpc error: code = NotFound desc = could not find container \"f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85\": container with ID starting with f69420c41df6617bab2330438f2fec25b2f1a59efb03ee8053ec8e7cb758af85 not found: ID does not exist"
Nov 02 01:42:49.634770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.634655    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-envoy" pod "548143a8-0c95-41f1-9c60-857747c3c4dd" (UID: "548143a8-0c95-41f1-9c60-857747c3c4dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:42:49.634770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.634744    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-podinfo" pod "548143a8-0c95-41f1-9c60-857747c3c4dd" (UID: "548143a8-0c95-41f1-9c60-857747c3c4dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:42:49.636341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.636270    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-kube-api-access-vqklt" pod "548143a8-0c95-41f1-9c60-857747c3c4dd" (UID: "548143a8-0c95-41f1-9c60-857747c3c4dd"). InnerVolumeSpecName "kube-api-access-vqklt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:49.719531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.719468    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.719735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.719708    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-metrics-certs\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.719837 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.719816    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9sp\" (UniqueName: \"kubernetes.io/projected/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-kube-api-access-dd9sp\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.719913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.719891    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-maas-api-tls\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.722613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.720055    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.722613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.720077    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.722613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.720091    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqklt\" (UniqueName: \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-kube-api-access-vqklt\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.722613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.720108    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.722613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.720123    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.722613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.720137    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548143a8-0c95-41f1-9c60-857747c3c4dd-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.722613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.720151    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548143a8-0c95-41f1-9c60-857747c3c4dd-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:49.722613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.720277    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.723693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.723671    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-maas-api-tls\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.723785 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.723752    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-metrics-certs\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.877198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.843168    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9sp\" (UniqueName: \"kubernetes.io/projected/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-kube-api-access-dd9sp\") pod \"maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.889972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.889935    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:49.965650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.961358    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:49.982104 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.982076    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"]
Nov 02 01:42:49.988732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:49.988706    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-9qx6z"]
Nov 02 01:42:50.024762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.024735    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6650af1a-9d41-48df-9df0-dafa1d85a2cc-kube-api-access-5rxk9\" (UniqueName: \"kubernetes.io/projected/6650af1a-9d41-48df-9df0-dafa1d85a2cc-kube-api-access-5rxk9\") pod \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") "
Nov 02 01:42:50.024918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.024781    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-xks-ca\") pod \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") "
Nov 02 01:42:50.024959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.024935    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-openshift-ca\") pod \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\" (UID: \"6650af1a-9d41-48df-9df0-dafa1d85a2cc\") "
Nov 02 01:42:50.025474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.025442    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-openshift-ca" pod "6650af1a-9d41-48df-9df0-dafa1d85a2cc" (UID: "6650af1a-9d41-48df-9df0-dafa1d85a2cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:42:50.026283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.026259    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:50.028267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.027734    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6650af1a-9d41-48df-9df0-dafa1d85a2cc-kube-api-access-5rxk9" pod "6650af1a-9d41-48df-9df0-dafa1d85a2cc" (UID: "6650af1a-9d41-48df-9df0-dafa1d85a2cc"). InnerVolumeSpecName "kube-api-access-5rxk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:42:50.028267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.027743    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-xks-ca" pod "6650af1a-9d41-48df-9df0-dafa1d85a2cc" (UID: "6650af1a-9d41-48df-9df0-dafa1d85a2cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:42:50.127742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.127658    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rxk9\" (UniqueName: \"kubernetes.io/projected/6650af1a-9d41-48df-9df0-dafa1d85a2cc-kube-api-access-5rxk9\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:50.127742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.127687    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6650af1a-9d41-48df-9df0-dafa1d85a2cc-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:42:50.204076 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:42:50.204036    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc9fe1d_e29c_4d82_919a_58d4d6e9364e.slice/crio-72c9dfb25ab8a3642787c437dcccd9e025d223a45f3d6eea039f854ce276642b WatchSource:0}: Error finding container 72c9dfb25ab8a3642787c437dcccd9e025d223a45f3d6eea039f854ce276642b: Status 404 returned error can't find the container with id 72c9dfb25ab8a3642787c437dcccd9e025d223a45f3d6eea039f854ce276642b
Nov 02 01:42:50.205193 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.205169    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"]
Nov 02 01:42:50.608719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.608680    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn" event={"ID":"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e","Type":"ContainerStarted","Data":"72c9dfb25ab8a3642787c437dcccd9e025d223a45f3d6eea039f854ce276642b"}
Nov 02 01:42:50.611436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.611412    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"
Nov 02 01:42:50.611436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.611424    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw" event={"ID":"6650af1a-9d41-48df-9df0-dafa1d85a2cc","Type":"ContainerDied","Data":"54b9339af7749998c60b82d6826988b1d5637aadffd285dbf2d968e03c54867b"}
Nov 02 01:42:50.611641 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.611471    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54b9339af7749998c60b82d6826988b1d5637aadffd285dbf2d968e03c54867b"
Nov 02 01:42:50.850092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:50.850059    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" path="/var/lib/kubelet/pods/548143a8-0c95-41f1-9c60-857747c3c4dd/volumes"
Nov 02 01:42:52.634619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:52.634537    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn" event={"ID":"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e","Type":"ContainerStarted","Data":"9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905"}
Nov 02 01:42:52.635051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:52.634633    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:52.661615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:52.661562    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn" podStartSLOduration=177.49232919 podStartE2EDuration="2m59.661545698s" totalImagesPullingTime="2.169216508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:52.656028331 +0000 UTC m=+1677.960961157" watchObservedRunningTime="2026-11-02 01:42:52.661545698 +0000 UTC m=+1677.966478538"
Nov 02 01:42:55.389047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.389006    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"]
Nov 02 01:42:55.389570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.389254    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy" containerID="cri-o://e285dfc421df0740b0f75a7cf666e3e6083642ba163514178fe873ead9903dee" gracePeriod=30
Nov 02 01:42:55.444722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.443227    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"]
Nov 02 01:42:55.444722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.443483    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy" containerID="cri-o://e0afa3651a8dd6ee13b0fdcf2f7f45566f8306f7a34cc81cf039b83a75db127b" gracePeriod=30
Nov 02 01:42:55.495674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.495640    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"]
Nov 02 01:42:55.496218 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.496183    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" podUID="8d13070b-3087-452c-a330-c5179e3bd0cf" containerName="istio-proxy" containerID="cri-o://0883378e5dd8828c38bf94c41b1a2dce7ae12490afbc2d90417f9574b50f2692" gracePeriod=30
Nov 02 01:42:55.722356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.722307    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"]
Nov 02 01:42:55.722582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.722558    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy" containerID="cri-o://c74e7d9b8d45a132b09303c041219fc71d7b8e8126c4a301f48e98a0259896d7" gracePeriod=30
Nov 02 01:42:55.872126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.871318    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"]
Nov 02 01:42:55.872126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.871615    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" podUID="39a26fed-b9e3-4085-9f05-d33b5532a9df" containerName="istio-proxy" containerID="cri-o://da16952a12914a9d6a8f09bbac314a86b7a1d5e837c1b8e64a93803e7c492ee1" gracePeriod=30
Nov 02 01:42:55.880391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.878838    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"]
Nov 02 01:42:55.880391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:55.879102    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" containerName="istio-proxy" containerID="cri-o://6ae2f75c860e68b4d05eb578a0e5644cc6721851df195f481db2f1e3aa728480" gracePeriod=30
Nov 02 01:42:57.052560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.049530    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"]
Nov 02 01:42:57.052560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.050511    4851 cpu_manager.go:414] "containerMap: removing container" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy"
Nov 02 01:42:57.052560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.050525    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548143a8-0c95-41f1-9c60-857747c3c4dd" containerName="istio-proxy"
Nov 02 01:42:57.052560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.050542    4851 cpu_manager.go:414] "containerMap: removing container" podUID="6650af1a-9d41-48df-9df0-dafa1d85a2cc" containerName="revoke-keys"
Nov 02 01:42:57.052560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.050548    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6650af1a-9d41-48df-9df0-dafa1d85a2cc" containerName="revoke-keys"
Nov 02 01:42:57.052560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.050709    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="6650af1a-9d41-48df-9df0-dafa1d85a2cc" containerName="revoke-keys"
Nov 02 01:42:57.058341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.055333    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"]
Nov 02 01:42:57.058341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.055412    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.062757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.062730    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:42:57.066862 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.066838    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:42:57.119581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.117806    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgnjn\" (UniqueName: \"kubernetes.io/projected/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-kube-api-access-wgnjn\") pod \"maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.120056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.120011    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.120479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.120454    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.253730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.253619    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgnjn\" (UniqueName: \"kubernetes.io/projected/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-kube-api-access-wgnjn\") pod \"maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.254019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.253984    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.254211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.254197    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.303418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.277677    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.346916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.346886    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgnjn\" (UniqueName: \"kubernetes.io/projected/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-kube-api-access-wgnjn\") pod \"maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.387711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.374044    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.555584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.555480    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:42:57.792027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.791993    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"]
Nov 02 01:42:57.799485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.799450    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:57.817792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.817704    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"]
Nov 02 01:42:57.856456 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:42:57.856422    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcaad55_ddb6_41a5_a0dc_77715b4efe1a.slice/crio-561155a1059f950301d099edf474faee5762681651e0489eeae4e44fadf1c1db WatchSource:0}: Error finding container 561155a1059f950301d099edf474faee5762681651e0489eeae4e44fadf1c1db: Status 404 returned error can't find the container with id 561155a1059f950301d099edf474faee5762681651e0489eeae4e44fadf1c1db
Nov 02 01:42:57.856953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.856907    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"]
Nov 02 01:42:57.868644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.868607    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338501e4-000f-4aed-92d7-97cae792d292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-befba964-wgl5g\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:57.868854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.868709    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338501e4-000f-4aed-92d7-97cae792d292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-befba964-wgl5g\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:57.868854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.868797    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfr7d\" (UniqueName: \"kubernetes.io/projected/338501e4-000f-4aed-92d7-97cae792d292-kube-api-access-nfr7d\") pod \"maas-api-revoke-sub-befba964-wgl5g\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:57.971092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.971060    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338501e4-000f-4aed-92d7-97cae792d292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-befba964-wgl5g\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:57.971272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.971121    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338501e4-000f-4aed-92d7-97cae792d292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-befba964-wgl5g\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:57.971272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.971186    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfr7d\" (UniqueName: \"kubernetes.io/projected/338501e4-000f-4aed-92d7-97cae792d292-kube-api-access-nfr7d\") pod \"maas-api-revoke-sub-befba964-wgl5g\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:57.971831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.971803    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338501e4-000f-4aed-92d7-97cae792d292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-befba964-wgl5g\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:57.973882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.973862    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338501e4-000f-4aed-92d7-97cae792d292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-befba964-wgl5g\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:57.984122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:57.984093    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfr7d\" (UniqueName: \"kubernetes.io/projected/338501e4-000f-4aed-92d7-97cae792d292-kube-api-access-nfr7d\") pod \"maas-api-revoke-sub-befba964-wgl5g\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:58.135167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:58.135089    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:42:58.584619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:58.580825    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"]
Nov 02 01:42:58.650003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:58.649976    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:42:58.706583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:58.706542    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs" event={"ID":"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a","Type":"ContainerStarted","Data":"90aa506e7c9b2e8639636359cc4be83abe9b36b95e616cf607e741a3cf281511"}
Nov 02 01:42:58.706837 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:58.706612    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs" event={"ID":"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a","Type":"ContainerStarted","Data":"561155a1059f950301d099edf474faee5762681651e0489eeae4e44fadf1c1db"}
Nov 02 01:42:58.708759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:58.708730    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g" event={"ID":"338501e4-000f-4aed-92d7-97cae792d292","Type":"ContainerStarted","Data":"5d436749103d0b547eee39732d2867a536b47e3d79342a12d4f5d25d5ca7c935"}
Nov 02 01:42:58.745503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:58.745439    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs" podStartSLOduration=2.745419234 podStartE2EDuration="2.745419234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:58.733928229 +0000 UTC m=+1684.038861068" watchObservedRunningTime="2026-11-02 01:42:58.745419234 +0000 UTC m=+1684.050352062"
Nov 02 01:42:59.481626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.481591    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"]
Nov 02 01:42:59.486598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.486578    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:42:59.498735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.498707    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"]
Nov 02 01:42:59.640237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.640153    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpzvh\" (UniqueName: \"kubernetes.io/projected/7a05937d-11ba-480a-8426-91ad623db8fa-kube-api-access-lpzvh\") pod \"maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:42:59.640613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.640583    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:42:59.640771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.640686    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:42:59.776348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.768631    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzvh\" (UniqueName: \"kubernetes.io/projected/7a05937d-11ba-480a-8426-91ad623db8fa-kube-api-access-lpzvh\") pod \"maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:42:59.776348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.769206    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:42:59.776348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.769352    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:42:59.776348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.775576    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:42:59.820318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.802465    4851 generic.go:356] "Generic (PLEG): container finished" podID="3bcaad55-ddb6-41a5-a0dc-77715b4efe1a" containerID="90aa506e7c9b2e8639636359cc4be83abe9b36b95e616cf607e741a3cf281511" exitCode=0
Nov 02 01:42:59.820318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.802591    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs" event={"ID":"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a","Type":"ContainerDied","Data":"90aa506e7c9b2e8639636359cc4be83abe9b36b95e616cf607e741a3cf281511"}
Nov 02 01:42:59.820318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.805871    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g" event={"ID":"338501e4-000f-4aed-92d7-97cae792d292","Type":"ContainerStarted","Data":"ca6dc05eaac96abb73ad41c33b5e4630aaa162b39dbdf384c045808db13f4836"}
Nov 02 01:42:59.820318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.810235    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzvh\" (UniqueName: \"kubernetes.io/projected/7a05937d-11ba-480a-8426-91ad623db8fa-kube-api-access-lpzvh\") pod \"maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:42:59.820318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.810451    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:42:59.866386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:42:59.866270    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g" podStartSLOduration=2.866254453 podStartE2EDuration="2.866254453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:59.85545781 +0000 UTC m=+1685.160390637" watchObservedRunningTime="2026-11-02 01:42:59.866254453 +0000 UTC m=+1685.171187329"
Nov 02 01:43:00.106241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.106180    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:43:00.241793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.241728    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main" containerID="cri-o://fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b" gracePeriod=45
Nov 02 01:43:00.397301 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:43:00.397255    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a05937d_11ba_480a_8426_91ad623db8fa.slice/crio-8fe226ad420b5e8adcf105d45a3418a52df9c5fadd7feb8c0fed47d1075bed17 WatchSource:0}: Error finding container 8fe226ad420b5e8adcf105d45a3418a52df9c5fadd7feb8c0fed47d1075bed17: Status 404 returned error can't find the container with id 8fe226ad420b5e8adcf105d45a3418a52df9c5fadd7feb8c0fed47d1075bed17
Nov 02 01:43:00.400005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.399912    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"]
Nov 02 01:43:00.656813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.656433    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:43:00.751330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.735958    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e1f2479-6898-45d6-8296-c68deb4b98bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1f2479-6898-45d6-8296-c68deb4b98bf-tls-certs\") pod \"6e1f2479-6898-45d6-8296-c68deb4b98bf\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") "
Nov 02 01:43:00.751330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.736041    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-home\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-home\") pod \"6e1f2479-6898-45d6-8296-c68deb4b98bf\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") "
Nov 02 01:43:00.751330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.736368    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-dshm\") pod \"6e1f2479-6898-45d6-8296-c68deb4b98bf\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") "
Nov 02 01:43:00.751330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.736464    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-tmp-dir\") pod \"6e1f2479-6898-45d6-8296-c68deb4b98bf\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") "
Nov 02 01:43:00.751330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.736582    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1f2479-6898-45d6-8296-c68deb4b98bf-kube-api-access-kd8kx\" (UniqueName: \"kubernetes.io/projected/6e1f2479-6898-45d6-8296-c68deb4b98bf-kube-api-access-kd8kx\") pod \"6e1f2479-6898-45d6-8296-c68deb4b98bf\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") "
Nov 02 01:43:00.751330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.736618    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-model-cache\") pod \"6e1f2479-6898-45d6-8296-c68deb4b98bf\" (UID: \"6e1f2479-6898-45d6-8296-c68deb4b98bf\") "
Nov 02 01:43:00.751330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.739604    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-home" pod "6e1f2479-6898-45d6-8296-c68deb4b98bf" (UID: "6e1f2479-6898-45d6-8296-c68deb4b98bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:00.751330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.741694    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-tmp-dir" pod "6e1f2479-6898-45d6-8296-c68deb4b98bf" (UID: "6e1f2479-6898-45d6-8296-c68deb4b98bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:00.751330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.741771    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:00.755775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.755718    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-dshm" pod "6e1f2479-6898-45d6-8296-c68deb4b98bf" (UID: "6e1f2479-6898-45d6-8296-c68deb4b98bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:00.758174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.756517    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-model-cache" pod "6e1f2479-6898-45d6-8296-c68deb4b98bf" (UID: "6e1f2479-6898-45d6-8296-c68deb4b98bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:00.769936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.767211    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1f2479-6898-45d6-8296-c68deb4b98bf-tls-certs" pod "6e1f2479-6898-45d6-8296-c68deb4b98bf" (UID: "6e1f2479-6898-45d6-8296-c68deb4b98bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:00.769936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.767390    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1f2479-6898-45d6-8296-c68deb4b98bf-kube-api-access-kd8kx" pod "6e1f2479-6898-45d6-8296-c68deb4b98bf" (UID: "6e1f2479-6898-45d6-8296-c68deb4b98bf"). InnerVolumeSpecName "kube-api-access-kd8kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:00.855041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.854688    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1f2479-6898-45d6-8296-c68deb4b98bf-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:00.855041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.854728    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:00.855041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.854742    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:00.855041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.854756    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd8kx\" (UniqueName: \"kubernetes.io/projected/6e1f2479-6898-45d6-8296-c68deb4b98bf-kube-api-access-kd8kx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:00.855041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.854773    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1f2479-6898-45d6-8296-c68deb4b98bf-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:00.918382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.911016    4851 generic.go:356] "Generic (PLEG): container finished" podID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerID="fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b" exitCode=0
Nov 02 01:43:00.918382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.911354    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"
Nov 02 01:43:00.922485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.922442    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm" event={"ID":"7a05937d-11ba-480a-8426-91ad623db8fa","Type":"ContainerStarted","Data":"8fe226ad420b5e8adcf105d45a3418a52df9c5fadd7feb8c0fed47d1075bed17"}
Nov 02 01:43:00.922626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.922520    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" event={"ID":"6e1f2479-6898-45d6-8296-c68deb4b98bf","Type":"ContainerDied","Data":"fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b"}
Nov 02 01:43:00.922626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.922543    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp" event={"ID":"6e1f2479-6898-45d6-8296-c68deb4b98bf","Type":"ContainerDied","Data":"9e0a432fa8b8305be346c12f1cd369e2ca318c89d03122aea42da85588de5866"}
Nov 02 01:43:00.922626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.922565    4851 scope.go:122] "RemoveContainer" containerID="fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b"
Nov 02 01:43:00.925210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.925185    4851 generic.go:356] "Generic (PLEG): container finished" podID="8d13070b-3087-452c-a330-c5179e3bd0cf" containerID="0883378e5dd8828c38bf94c41b1a2dce7ae12490afbc2d90417f9574b50f2692" exitCode=0
Nov 02 01:43:00.925358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.925284    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" event={"ID":"8d13070b-3087-452c-a330-c5179e3bd0cf","Type":"ContainerDied","Data":"0883378e5dd8828c38bf94c41b1a2dce7ae12490afbc2d90417f9574b50f2692"}
Nov 02 01:43:00.927246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.927222    4851 generic.go:356] "Generic (PLEG): container finished" podID="338501e4-000f-4aed-92d7-97cae792d292" containerID="ca6dc05eaac96abb73ad41c33b5e4630aaa162b39dbdf384c045808db13f4836" exitCode=0
Nov 02 01:43:00.927413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.927322    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g" event={"ID":"338501e4-000f-4aed-92d7-97cae792d292","Type":"ContainerDied","Data":"ca6dc05eaac96abb73ad41c33b5e4630aaa162b39dbdf384c045808db13f4836"}
Nov 02 01:43:00.938367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.933812    4851 generic.go:356] "Generic (PLEG): container finished" podID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerID="e285dfc421df0740b0f75a7cf666e3e6083642ba163514178fe873ead9903dee" exitCode=0
Nov 02 01:43:00.938367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.933904    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" event={"ID":"375a3837-853f-4bb4-b078-b2116e0d80f0","Type":"ContainerDied","Data":"e285dfc421df0740b0f75a7cf666e3e6083642ba163514178fe873ead9903dee"}
Nov 02 01:43:00.938367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.937472    4851 generic.go:356] "Generic (PLEG): container finished" podID="dac1c433-8412-47ae-9794-ce298c3593bd" containerID="e0afa3651a8dd6ee13b0fdcf2f7f45566f8306f7a34cc81cf039b83a75db127b" exitCode=0
Nov 02 01:43:00.938367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:00.937605    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" event={"ID":"dac1c433-8412-47ae-9794-ce298c3593bd","Type":"ContainerDied","Data":"e0afa3651a8dd6ee13b0fdcf2f7f45566f8306f7a34cc81cf039b83a75db127b"}
Nov 02 01:43:01.008844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.008605    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"]
Nov 02 01:43:01.021586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.021485    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-addad9/test-model-addad9-kserve-cbcb4cd8f-l5kkp"]
Nov 02 01:43:01.127076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.126645    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:43:01.146666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.146490    4851 scope.go:122] "RemoveContainer" containerID="fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b"
Nov 02 01:43:01.147514 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:01.147418    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b\": container with ID starting with fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b not found: ID does not exist" containerID="fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b"
Nov 02 01:43:01.147514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.147454    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b"} err="failed to get container status \"fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b\": rpc error: code = NotFound desc = could not find container \"fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b\": container with ID starting with fce5eab9e4f7f355ce998bec3a0b07893b3030837990795fb382de006f8b8f6b not found: ID does not exist"
Nov 02 01:43:01.254971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.254887    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:43:01.259164 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.259130    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:43:01.315679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.315648    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-istio-token\" (UniqueName: \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-istio-token\") pod \"dac1c433-8412-47ae-9794-ce298c3593bd\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") "
Nov 02 01:43:01.322709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.322670    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dac1c433-8412-47ae-9794-ce298c3593bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac1c433-8412-47ae-9794-ce298c3593bd-istio-podinfo\") pod \"dac1c433-8412-47ae-9794-ce298c3593bd\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") "
Nov 02 01:43:01.324418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.324394    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-data\") pod \"dac1c433-8412-47ae-9794-ce298c3593bd\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") "
Nov 02 01:43:01.324641 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.324620    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-data" pod "dac1c433-8412-47ae-9794-ce298c3593bd" (UID: "dac1c433-8412-47ae-9794-ce298c3593bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.325618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.325597    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-socket\") pod \"dac1c433-8412-47ae-9794-ce298c3593bd\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") "
Nov 02 01:43:01.326075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.325972    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-socket" pod "dac1c433-8412-47ae-9794-ce298c3593bd" (UID: "dac1c433-8412-47ae-9794-ce298c3593bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.326590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.326564    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-certs\") pod \"dac1c433-8412-47ae-9794-ce298c3593bd\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") "
Nov 02 01:43:01.327050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.326948    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-certs" pod "dac1c433-8412-47ae-9794-ce298c3593bd" (UID: "dac1c433-8412-47ae-9794-ce298c3593bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.327420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.327270    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-envoy\") pod \"dac1c433-8412-47ae-9794-ce298c3593bd\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") "
Nov 02 01:43:01.329010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.328208    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-credential-socket\") pod \"dac1c433-8412-47ae-9794-ce298c3593bd\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") "
Nov 02 01:43:01.329224 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.328972    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-credential-socket" pod "dac1c433-8412-47ae-9794-ce298c3593bd" (UID: "dac1c433-8412-47ae-9794-ce298c3593bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.329403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.329386    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-kube-api-access-bmwxp\" (UniqueName: \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-kube-api-access-bmwxp\") pod \"dac1c433-8412-47ae-9794-ce298c3593bd\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") "
Nov 02 01:43:01.330386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.329917    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac1c433-8412-47ae-9794-ce298c3593bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac1c433-8412-47ae-9794-ce298c3593bd-istiod-ca-cert\") pod \"dac1c433-8412-47ae-9794-ce298c3593bd\" (UID: \"dac1c433-8412-47ae-9794-ce298c3593bd\") "
Nov 02 01:43:01.330772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.330748    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac1c433-8412-47ae-9794-ce298c3593bd-istiod-ca-cert" pod "dac1c433-8412-47ae-9794-ce298c3593bd" (UID: "dac1c433-8412-47ae-9794-ce298c3593bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:01.333939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.333916    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-kube-api-access-bmwxp" pod "dac1c433-8412-47ae-9794-ce298c3593bd" (UID: "dac1c433-8412-47ae-9794-ce298c3593bd"). InnerVolumeSpecName "kube-api-access-bmwxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.334106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.334087    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-envoy" pod "dac1c433-8412-47ae-9794-ce298c3593bd" (UID: "dac1c433-8412-47ae-9794-ce298c3593bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.335106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.335083    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.336260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.336241    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmwxp\" (UniqueName: \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-kube-api-access-bmwxp\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.336733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.336717    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac1c433-8412-47ae-9794-ce298c3593bd-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.337559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.336837    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.337559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.336855    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.337559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.336868    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.337559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.336882    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac1c433-8412-47ae-9794-ce298c3593bd-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.347121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.342409    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-istio-token" pod "dac1c433-8412-47ae-9794-ce298c3593bd" (UID: "dac1c433-8412-47ae-9794-ce298c3593bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.347121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.344452    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dac1c433-8412-47ae-9794-ce298c3593bd-istio-podinfo" pod "dac1c433-8412-47ae-9794-ce298c3593bd" (UID: "dac1c433-8412-47ae-9794-ce298c3593bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:01.402722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.402682    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:43:01.420275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.420237    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:43:01.449102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.449071    4851 scope.go:122] "RemoveContainer" containerID="0883378e5dd8828c38bf94c41b1a2dce7ae12490afbc2d90417f9574b50f2692"
Nov 02 01:43:01.450809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.450775    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-credential-socket\") pod \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") "
Nov 02 01:43:01.451402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.451383    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-token\" (UniqueName: \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-token\") pod \"375a3837-853f-4bb4-b078-b2116e0d80f0\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") "
Nov 02 01:43:01.451568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.451553    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-data\") pod \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") "
Nov 02 01:43:01.451706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.451692    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-envoy\") pod \"8d13070b-3087-452c-a330-c5179e3bd0cf\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") "
Nov 02 01:43:01.451857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.451842    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-credential-socket\") pod \"8d13070b-3087-452c-a330-c5179e3bd0cf\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") "
Nov 02 01:43:01.451987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.451973    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-token\" (UniqueName: \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-token\") pod \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") "
Nov 02 01:43:01.452086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.452074    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-envoy\") pod \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") "
Nov 02 01:43:01.452183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.452165    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-socket\") pod \"8d13070b-3087-452c-a330-c5179e3bd0cf\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") "
Nov 02 01:43:01.452309 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.452277    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-xks-ca\") pod \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") "
Nov 02 01:43:01.452461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.452446    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-socket\") pod \"375a3837-853f-4bb4-b078-b2116e0d80f0\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") "
Nov 02 01:43:01.452625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.452598    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-credential-socket" pod "93c39b84-1325-4b9f-8cbf-ef64ecf31237" (UID: "93c39b84-1325-4b9f-8cbf-ef64ecf31237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.452751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.452733    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-token\" (UniqueName: \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-token\") pod \"8d13070b-3087-452c-a330-c5179e3bd0cf\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") "
Nov 02 01:43:01.452949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.452931    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-kube-api-access-gdg54\" (UniqueName: \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-kube-api-access-gdg54\") pod \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") "
Nov 02 01:43:01.453246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.453231    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-certs\") pod \"8d13070b-3087-452c-a330-c5179e3bd0cf\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") "
Nov 02 01:43:01.453415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.453402    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-credential-socket\") pod \"375a3837-853f-4bb4-b078-b2116e0d80f0\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") "
Nov 02 01:43:01.453527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.453515    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-kube-api-access-jrlt5\" (UniqueName: \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-kube-api-access-jrlt5\") pod \"375a3837-853f-4bb4-b078-b2116e0d80f0\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") "
Nov 02 01:43:01.453648 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.453636    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-certs\") pod \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") "
Nov 02 01:43:01.454978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.454957    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-podinfo\") pod \"8d13070b-3087-452c-a330-c5179e3bd0cf\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") "
Nov 02 01:43:01.455144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.455128    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-podinfo\") pod \"375a3837-853f-4bb4-b078-b2116e0d80f0\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") "
Nov 02 01:43:01.455279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.455262    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-kube-api-access-5mcz8\" (UniqueName: \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-kube-api-access-5mcz8\") pod \"8d13070b-3087-452c-a330-c5179e3bd0cf\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") "
Nov 02 01:43:01.456397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.456377    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-kube-api-access-wgnjn\" (UniqueName: \"kubernetes.io/projected/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-kube-api-access-wgnjn\") pod \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") "
Nov 02 01:43:01.457359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.457280    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d13070b-3087-452c-a330-c5179e3bd0cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d13070b-3087-452c-a330-c5179e3bd0cf-istiod-ca-cert\") pod \"8d13070b-3087-452c-a330-c5179e3bd0cf\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") "
Nov 02 01:43:01.457503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.457378    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375a3837-853f-4bb4-b078-b2116e0d80f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375a3837-853f-4bb4-b078-b2116e0d80f0-istiod-ca-cert\") pod \"375a3837-853f-4bb4-b078-b2116e0d80f0\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") "
Nov 02 01:43:01.457503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.457430    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-envoy\") pod \"375a3837-853f-4bb4-b078-b2116e0d80f0\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") "
Nov 02 01:43:01.457503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.457463    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-openshift-ca\") pod \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\" (UID: \"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a\") "
Nov 02 01:43:01.457686 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.457536    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istiod-ca-cert\") pod \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") "
Nov 02 01:43:01.457686 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.457582    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-certs\") pod \"375a3837-853f-4bb4-b078-b2116e0d80f0\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") "
Nov 02 01:43:01.457686 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.457616    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-data\") pod \"375a3837-853f-4bb4-b078-b2116e0d80f0\" (UID: \"375a3837-853f-4bb4-b078-b2116e0d80f0\") "
Nov 02 01:43:01.457686 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.457645    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-podinfo\") pod \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") "
Nov 02 01:43:01.457916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.457760    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-socket\") pod \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\" (UID: \"93c39b84-1325-4b9f-8cbf-ef64ecf31237\") "
Nov 02 01:43:01.457916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.457832    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-data\") pod \"8d13070b-3087-452c-a330-c5179e3bd0cf\" (UID: \"8d13070b-3087-452c-a330-c5179e3bd0cf\") "
Nov 02 01:43:01.459009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.458987    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac1c433-8412-47ae-9794-ce298c3593bd-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.459134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.459014    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac1c433-8412-47ae-9794-ce298c3593bd-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.459134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.459028    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.453939    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-certs" pod "93c39b84-1325-4b9f-8cbf-ef64ecf31237" (UID: "93c39b84-1325-4b9f-8cbf-ef64ecf31237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.454062    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-credential-socket" pod "8d13070b-3087-452c-a330-c5179e3bd0cf" (UID: "8d13070b-3087-452c-a330-c5179e3bd0cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.454549    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-data" pod "93c39b84-1325-4b9f-8cbf-ef64ecf31237" (UID: "93c39b84-1325-4b9f-8cbf-ef64ecf31237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.456463    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-socket" pod "375a3837-853f-4bb4-b078-b2116e0d80f0" (UID: "375a3837-853f-4bb4-b078-b2116e0d80f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.458676    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-socket" pod "8d13070b-3087-452c-a330-c5179e3bd0cf" (UID: "8d13070b-3087-452c-a330-c5179e3bd0cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.459473    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istiod-ca-cert" pod "93c39b84-1325-4b9f-8cbf-ef64ecf31237" (UID: "93c39b84-1325-4b9f-8cbf-ef64ecf31237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.459740    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d13070b-3087-452c-a330-c5179e3bd0cf-istiod-ca-cert" pod "8d13070b-3087-452c-a330-c5179e3bd0cf" (UID: "8d13070b-3087-452c-a330-c5179e3bd0cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.460023    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-openshift-ca" pod "3bcaad55-ddb6-41a5-a0dc-77715b4efe1a" (UID: "3bcaad55-ddb6-41a5-a0dc-77715b4efe1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.460725    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375a3837-853f-4bb4-b078-b2116e0d80f0-istiod-ca-cert" pod "375a3837-853f-4bb4-b078-b2116e0d80f0" (UID: "375a3837-853f-4bb4-b078-b2116e0d80f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.463220    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-certs" pod "8d13070b-3087-452c-a330-c5179e3bd0cf" (UID: "8d13070b-3087-452c-a330-c5179e3bd0cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.463359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.463248    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-data" pod "8d13070b-3087-452c-a330-c5179e3bd0cf" (UID: "8d13070b-3087-452c-a330-c5179e3bd0cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.463995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.463429    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-socket" pod "93c39b84-1325-4b9f-8cbf-ef64ecf31237" (UID: "93c39b84-1325-4b9f-8cbf-ef64ecf31237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.473926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.473172    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-certs" pod "375a3837-853f-4bb4-b078-b2116e0d80f0" (UID: "375a3837-853f-4bb4-b078-b2116e0d80f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.480840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.480794    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-credential-socket" pod "375a3837-853f-4bb4-b078-b2116e0d80f0" (UID: "375a3837-853f-4bb4-b078-b2116e0d80f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.481636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.481611    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-data" pod "375a3837-853f-4bb4-b078-b2116e0d80f0" (UID: "375a3837-853f-4bb4-b078-b2116e0d80f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.486653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.486488    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-podinfo" pod "93c39b84-1325-4b9f-8cbf-ef64ecf31237" (UID: "93c39b84-1325-4b9f-8cbf-ef64ecf31237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:01.486653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.486529    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-podinfo" pod "375a3837-853f-4bb4-b078-b2116e0d80f0" (UID: "375a3837-853f-4bb4-b078-b2116e0d80f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:01.486653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.486616    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-kube-api-access-wgnjn" pod "3bcaad55-ddb6-41a5-a0dc-77715b4efe1a" (UID: "3bcaad55-ddb6-41a5-a0dc-77715b4efe1a"). InnerVolumeSpecName "kube-api-access-wgnjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.493577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.486932    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-kube-api-access-jrlt5" pod "375a3837-853f-4bb4-b078-b2116e0d80f0" (UID: "375a3837-853f-4bb4-b078-b2116e0d80f0"). InnerVolumeSpecName "kube-api-access-jrlt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.496190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.495803    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-token" pod "93c39b84-1325-4b9f-8cbf-ef64ecf31237" (UID: "93c39b84-1325-4b9f-8cbf-ef64ecf31237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.496190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.495911    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-token" pod "375a3837-853f-4bb4-b078-b2116e0d80f0" (UID: "375a3837-853f-4bb4-b078-b2116e0d80f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.496190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.495972    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-xks-ca" pod "3bcaad55-ddb6-41a5-a0dc-77715b4efe1a" (UID: "3bcaad55-ddb6-41a5-a0dc-77715b4efe1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:01.496190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.496024    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-podinfo" pod "8d13070b-3087-452c-a330-c5179e3bd0cf" (UID: "8d13070b-3087-452c-a330-c5179e3bd0cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:01.496190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.496078    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-kube-api-access-gdg54" pod "93c39b84-1325-4b9f-8cbf-ef64ecf31237" (UID: "93c39b84-1325-4b9f-8cbf-ef64ecf31237"). InnerVolumeSpecName "kube-api-access-gdg54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.496190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.496124    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-kube-api-access-5mcz8" pod "8d13070b-3087-452c-a330-c5179e3bd0cf" (UID: "8d13070b-3087-452c-a330-c5179e3bd0cf"). InnerVolumeSpecName "kube-api-access-5mcz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.496190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.496169    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-token" pod "8d13070b-3087-452c-a330-c5179e3bd0cf" (UID: "8d13070b-3087-452c-a330-c5179e3bd0cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.508663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.508599    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-envoy" pod "93c39b84-1325-4b9f-8cbf-ef64ecf31237" (UID: "93c39b84-1325-4b9f-8cbf-ef64ecf31237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.508963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.508900    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-envoy" pod "8d13070b-3087-452c-a330-c5179e3bd0cf" (UID: "8d13070b-3087-452c-a330-c5179e3bd0cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.512348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.512070    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-envoy" pod "375a3837-853f-4bb4-b078-b2116e0d80f0" (UID: "375a3837-853f-4bb4-b078-b2116e0d80f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.519391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.519228    4851 scope.go:122] "RemoveContainer" containerID="e285dfc421df0740b0f75a7cf666e3e6083642ba163514178fe873ead9903dee"
Nov 02 01:43:01.552373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.552346    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:43:01.553680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.553668    4851 scope.go:122] "RemoveContainer" containerID="e0afa3651a8dd6ee13b0fdcf2f7f45566f8306f7a34cc81cf039b83a75db127b"
Nov 02 01:43:01.560832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560793    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdg54\" (UniqueName: \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-kube-api-access-gdg54\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.560832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560821    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.560832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560832    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560841    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrlt5\" (UniqueName: \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-kube-api-access-jrlt5\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560851    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560859    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560867    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560877    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mcz8\" (UniqueName: \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-kube-api-access-5mcz8\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560886    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgnjn\" (UniqueName: \"kubernetes.io/projected/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-kube-api-access-wgnjn\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560894    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d13070b-3087-452c-a330-c5179e3bd0cf-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560903    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/375a3837-853f-4bb4-b078-b2116e0d80f0-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560911    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560920    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560928    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560937    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560955    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560964    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560974    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.560988    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.561000    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/375a3837-853f-4bb4-b078-b2116e0d80f0-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.561011    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.561019    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.561027    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.561036    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.561049    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c39b84-1325-4b9f-8cbf-ef64ecf31237-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.561061    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d13070b-3087-452c-a330-c5179e3bd0cf-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.561073    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.561085    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/375a3837-853f-4bb4-b078-b2116e0d80f0-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.561142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.561098    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d13070b-3087-452c-a330-c5179e3bd0cf-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.597200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.597176    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:43:01.602242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.601976    4851 scope.go:122] "RemoveContainer" containerID="27e698823a47da5ebc17e07a1917d06213c78105abad3babd774dd02a1226905"
Nov 02 01:43:01.619020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.618992    4851 scope.go:122] "RemoveContainer" containerID="da16952a12914a9d6a8f09bbac314a86b7a1d5e837c1b8e64a93803e7c492ee1"
Nov 02 01:43:01.645011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.644981    4851 scope.go:122] "RemoveContainer" containerID="b68d48cf98a0c88d1052781961474a3cda98e2a273a98fce62672a7a0f898d1a"
Nov 02 01:43:01.661122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.661088    4851 scope.go:122] "RemoveContainer" containerID="c74e7d9b8d45a132b09303c041219fc71d7b8e8126c4a301f48e98a0259896d7"
Nov 02 01:43:01.661757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.661735    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-kube-api-access-fphq9\" (UniqueName: \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-kube-api-access-fphq9\") pod \"39a26fed-b9e3-4085-9f05-d33b5532a9df\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") "
Nov 02 01:43:01.661844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.661785    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-credential-socket\") pod \"39a26fed-b9e3-4085-9f05-d33b5532a9df\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") "
Nov 02 01:43:01.661844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.661829    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-data\") pod \"1d1aa882-4253-4421-a52c-d300b159fb5c\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") "
Nov 02 01:43:01.661968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.661864    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-data\") pod \"39a26fed-b9e3-4085-9f05-d33b5532a9df\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") "
Nov 02 01:43:01.661968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.661897    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-certs\") pod \"39a26fed-b9e3-4085-9f05-d33b5532a9df\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") "
Nov 02 01:43:01.661968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.661948    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-socket\") pod \"39a26fed-b9e3-4085-9f05-d33b5532a9df\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") "
Nov 02 01:43:01.662133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662023    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-envoy\") pod \"1d1aa882-4253-4421-a52c-d300b159fb5c\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") "
Nov 02 01:43:01.662133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662091    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-envoy\") pod \"39a26fed-b9e3-4085-9f05-d33b5532a9df\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") "
Nov 02 01:43:01.662133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662122    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1aa882-4253-4421-a52c-d300b159fb5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1aa882-4253-4421-a52c-d300b159fb5c-istiod-ca-cert\") pod \"1d1aa882-4253-4421-a52c-d300b159fb5c\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") "
Nov 02 01:43:01.662336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662176    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-podinfo\") pod \"39a26fed-b9e3-4085-9f05-d33b5532a9df\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") "
Nov 02 01:43:01.662336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662263    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-socket\") pod \"1d1aa882-4253-4421-a52c-d300b159fb5c\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") "
Nov 02 01:43:01.662336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662311    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-podinfo\") pod \"1d1aa882-4253-4421-a52c-d300b159fb5c\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") "
Nov 02 01:43:01.662508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662416    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a26fed-b9e3-4085-9f05-d33b5532a9df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a26fed-b9e3-4085-9f05-d33b5532a9df-istiod-ca-cert\") pod \"39a26fed-b9e3-4085-9f05-d33b5532a9df\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") "
Nov 02 01:43:01.662508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662465    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-certs\") pod \"1d1aa882-4253-4421-a52c-d300b159fb5c\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") "
Nov 02 01:43:01.662508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662494    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-token\" (UniqueName: \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-token\") pod \"39a26fed-b9e3-4085-9f05-d33b5532a9df\" (UID: \"39a26fed-b9e3-4085-9f05-d33b5532a9df\") "
Nov 02 01:43:01.662670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662555    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-kube-api-access-jcqdq\" (UniqueName: \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-kube-api-access-jcqdq\") pod \"1d1aa882-4253-4421-a52c-d300b159fb5c\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") "
Nov 02 01:43:01.662670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662583    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-token\" (UniqueName: \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-token\") pod \"1d1aa882-4253-4421-a52c-d300b159fb5c\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") "
Nov 02 01:43:01.662670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662653    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-socket" pod "39a26fed-b9e3-4085-9f05-d33b5532a9df" (UID: "39a26fed-b9e3-4085-9f05-d33b5532a9df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.663505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662719    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-credential-socket\") pod \"1d1aa882-4253-4421-a52c-d300b159fb5c\" (UID: \"1d1aa882-4253-4421-a52c-d300b159fb5c\") "
Nov 02 01:43:01.663505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.662803    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-credential-socket" pod "39a26fed-b9e3-4085-9f05-d33b5532a9df" (UID: "39a26fed-b9e3-4085-9f05-d33b5532a9df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.663505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.663021    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-socket" pod "1d1aa882-4253-4421-a52c-d300b159fb5c" (UID: "1d1aa882-4253-4421-a52c-d300b159fb5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.663779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.663671    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1aa882-4253-4421-a52c-d300b159fb5c-istiod-ca-cert" pod "1d1aa882-4253-4421-a52c-d300b159fb5c" (UID: "1d1aa882-4253-4421-a52c-d300b159fb5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.663839    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-certs" pod "39a26fed-b9e3-4085-9f05-d33b5532a9df" (UID: "39a26fed-b9e3-4085-9f05-d33b5532a9df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664084    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-data" pod "39a26fed-b9e3-4085-9f05-d33b5532a9df" (UID: "39a26fed-b9e3-4085-9f05-d33b5532a9df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664240    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-certs" pod "1d1aa882-4253-4421-a52c-d300b159fb5c" (UID: "1d1aa882-4253-4421-a52c-d300b159fb5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664177    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a26fed-b9e3-4085-9f05-d33b5532a9df-istiod-ca-cert" pod "39a26fed-b9e3-4085-9f05-d33b5532a9df" (UID: "39a26fed-b9e3-4085-9f05-d33b5532a9df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664396    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-data" pod "1d1aa882-4253-4421-a52c-d300b159fb5c" (UID: "1d1aa882-4253-4421-a52c-d300b159fb5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664413    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-credential-socket" pod "1d1aa882-4253-4421-a52c-d300b159fb5c" (UID: "1d1aa882-4253-4421-a52c-d300b159fb5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664598    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-kube-api-access-fphq9" pod "39a26fed-b9e3-4085-9f05-d33b5532a9df" (UID: "39a26fed-b9e3-4085-9f05-d33b5532a9df"). InnerVolumeSpecName "kube-api-access-fphq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664735    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a26fed-b9e3-4085-9f05-d33b5532a9df-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664758    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664774    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664789    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664803    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664816    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664830    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664844    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664857    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1aa882-4253-4421-a52c-d300b159fb5c-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.664893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.664871    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.666047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.665767    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-envoy" pod "1d1aa882-4253-4421-a52c-d300b159fb5c" (UID: "1d1aa882-4253-4421-a52c-d300b159fb5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.666785 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.666759    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-token" pod "1d1aa882-4253-4421-a52c-d300b159fb5c" (UID: "1d1aa882-4253-4421-a52c-d300b159fb5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.667935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.667910    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-podinfo" pod "39a26fed-b9e3-4085-9f05-d33b5532a9df" (UID: "39a26fed-b9e3-4085-9f05-d33b5532a9df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:01.668119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.668096    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-podinfo" pod "1d1aa882-4253-4421-a52c-d300b159fb5c" (UID: "1d1aa882-4253-4421-a52c-d300b159fb5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:01.668404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.668381    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-envoy" pod "39a26fed-b9e3-4085-9f05-d33b5532a9df" (UID: "39a26fed-b9e3-4085-9f05-d33b5532a9df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:01.669031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.669010    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-kube-api-access-jcqdq" pod "1d1aa882-4253-4421-a52c-d300b159fb5c" (UID: "1d1aa882-4253-4421-a52c-d300b159fb5c"). InnerVolumeSpecName "kube-api-access-jcqdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.669889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.669867    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-token" pod "39a26fed-b9e3-4085-9f05-d33b5532a9df" (UID: "39a26fed-b9e3-4085-9f05-d33b5532a9df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:01.686233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.686099    4851 scope.go:122] "RemoveContainer" containerID="6ae2f75c860e68b4d05eb578a0e5644cc6721851df195f481db2f1e3aa728480"
Nov 02 01:43:01.789656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.786923    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.791103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.791053    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.791391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.791369    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcqdq\" (UniqueName: \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-kube-api-access-jcqdq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.791621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.791535    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.791621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.791559    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fphq9\" (UniqueName: \"kubernetes.io/projected/39a26fed-b9e3-4085-9f05-d33b5532a9df-kube-api-access-fphq9\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.791621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.791574    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1aa882-4253-4421-a52c-d300b159fb5c-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.791621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.791587    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.791621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.791600    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a26fed-b9e3-4085-9f05-d33b5532a9df-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:01.949397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.949371    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"
Nov 02 01:43:01.949540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.949414    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs" event={"ID":"3bcaad55-ddb6-41a5-a0dc-77715b4efe1a","Type":"ContainerDied","Data":"561155a1059f950301d099edf474faee5762681651e0489eeae4e44fadf1c1db"}
Nov 02 01:43:01.949540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.949464    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561155a1059f950301d099edf474faee5762681651e0489eeae4e44fadf1c1db"
Nov 02 01:43:01.951033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.951017    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"
Nov 02 01:43:01.951033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.951046    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" event={"ID":"1d1aa882-4253-4421-a52c-d300b159fb5c","Type":"ContainerDied","Data":"6ae2f75c860e68b4d05eb578a0e5644cc6721851df195f481db2f1e3aa728480"}
Nov 02 01:43:01.951599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.951120    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p" event={"ID":"1d1aa882-4253-4421-a52c-d300b159fb5c","Type":"ContainerDied","Data":"382f44a8800f8523a1d255b7ff4514887cb9b5278b091d6b402cf54693a962dd"}
Nov 02 01:43:01.951599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.951146    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" event={"ID":"93c39b84-1325-4b9f-8cbf-ef64ecf31237","Type":"ContainerDied","Data":"c74e7d9b8d45a132b09303c041219fc71d7b8e8126c4a301f48e98a0259896d7"}
Nov 02 01:43:01.951599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.951172    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm" event={"ID":"93c39b84-1325-4b9f-8cbf-ef64ecf31237","Type":"ContainerDied","Data":"97a0f30d0b4c7882b9d467fd49b8df59c69c5f3eac036d06ae5dbba794298889"}
Nov 02 01:43:01.951599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.951218    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7" event={"ID":"8d13070b-3087-452c-a330-c5179e3bd0cf","Type":"ContainerDied","Data":"62aa3e49af6ba33fd005289603625f8d94778eeb7741855864a660dc69af6110"}
Nov 02 01:43:01.951599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.951243    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" event={"ID":"39a26fed-b9e3-4085-9f05-d33b5532a9df","Type":"ContainerDied","Data":"da16952a12914a9d6a8f09bbac314a86b7a1d5e837c1b8e64a93803e7c492ee1"}
Nov 02 01:43:01.951599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.951283    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69" event={"ID":"39a26fed-b9e3-4085-9f05-d33b5532a9df","Type":"ContainerDied","Data":"6e97d1ee287654e729798c2eafc72341d8f6ee54b4f602f75b533dd3546e0b19"}
Nov 02 01:43:01.951599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.951357    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc" event={"ID":"375a3837-853f-4bb4-b078-b2116e0d80f0","Type":"ContainerDied","Data":"0796d1a3342b6433d7b330beb7980336699114634c7b247d2485cd7c79895c63"}
Nov 02 01:43:01.951599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.951376    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6" event={"ID":"dac1c433-8412-47ae-9794-ce298c3593bd","Type":"ContainerDied","Data":"cddbfa25d3d589c2832cf95a23ad4f79a6273206eff9b41918f0f06a73807d25"}
Nov 02 01:43:01.953204 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.953171    4851 generic.go:356] "Generic (PLEG): container finished" podID="7a05937d-11ba-480a-8426-91ad623db8fa" containerID="edc4b3f64d500ef7d10dc8bfed212869032c42dca873357a9f0942d7cc98f005" exitCode=0
Nov 02 01:43:01.953370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.953247    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm" event={"ID":"7a05937d-11ba-480a-8426-91ad623db8fa","Type":"ContainerDied","Data":"edc4b3f64d500ef7d10dc8bfed212869032c42dca873357a9f0942d7cc98f005"}
Nov 02 01:43:01.953370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.953362    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"
Nov 02 01:43:01.953664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.953649    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"
Nov 02 01:43:01.953969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.953951    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"
Nov 02 01:43:01.954382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.954359    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"
Nov 02 01:43:01.954661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:01.954608    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"
Nov 02 01:43:02.152188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.152159    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"]
Nov 02 01:43:02.165869 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.165794    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-xc9h6"]
Nov 02 01:43:02.174022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.173995    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:43:02.176179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.176158    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"]
Nov 02 01:43:02.181301 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.181260    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-2rsjc"]
Nov 02 01:43:02.189655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.189634    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"]
Nov 02 01:43:02.197802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.197782    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-vxr69"]
Nov 02 01:43:02.210734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.210709    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"]
Nov 02 01:43:02.215405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.215383    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-jx45p"]
Nov 02 01:43:02.227335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.227264    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"]
Nov 02 01:43:02.231796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.231775    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-mpqrm"]
Nov 02 01:43:02.243132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.243108    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"]
Nov 02 01:43:02.250168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.250146    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-swjv7"]
Nov 02 01:43:02.300230 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.300195    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338501e4-000f-4aed-92d7-97cae792d292-kube-api-access-nfr7d\" (UniqueName: \"kubernetes.io/projected/338501e4-000f-4aed-92d7-97cae792d292-kube-api-access-nfr7d\") pod \"338501e4-000f-4aed-92d7-97cae792d292\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") "
Nov 02 01:43:02.300579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.300552    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338501e4-000f-4aed-92d7-97cae792d292-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338501e4-000f-4aed-92d7-97cae792d292-maas-api-openshift-ca\") pod \"338501e4-000f-4aed-92d7-97cae792d292\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") "
Nov 02 01:43:02.300709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.300698    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/338501e4-000f-4aed-92d7-97cae792d292-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338501e4-000f-4aed-92d7-97cae792d292-maas-api-xks-ca\") pod \"338501e4-000f-4aed-92d7-97cae792d292\" (UID: \"338501e4-000f-4aed-92d7-97cae792d292\") "
Nov 02 01:43:02.301507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.301476    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338501e4-000f-4aed-92d7-97cae792d292-maas-api-openshift-ca" pod "338501e4-000f-4aed-92d7-97cae792d292" (UID: "338501e4-000f-4aed-92d7-97cae792d292"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:02.303197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.303171    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/338501e4-000f-4aed-92d7-97cae792d292-maas-api-xks-ca" pod "338501e4-000f-4aed-92d7-97cae792d292" (UID: "338501e4-000f-4aed-92d7-97cae792d292"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:02.303357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.303241    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338501e4-000f-4aed-92d7-97cae792d292-kube-api-access-nfr7d" pod "338501e4-000f-4aed-92d7-97cae792d292" (UID: "338501e4-000f-4aed-92d7-97cae792d292"). InnerVolumeSpecName "kube-api-access-nfr7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:02.402976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.402940    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338501e4-000f-4aed-92d7-97cae792d292-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:02.402976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.402970    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338501e4-000f-4aed-92d7-97cae792d292-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:02.402976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.402981    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfr7d\" (UniqueName: \"kubernetes.io/projected/338501e4-000f-4aed-92d7-97cae792d292-kube-api-access-nfr7d\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:02.851443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.851399    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" path="/var/lib/kubelet/pods/1d1aa882-4253-4421-a52c-d300b159fb5c/volumes"
Nov 02 01:43:02.852972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.852948    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" path="/var/lib/kubelet/pods/375a3837-853f-4bb4-b078-b2116e0d80f0/volumes"
Nov 02 01:43:02.853971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.853950    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a26fed-b9e3-4085-9f05-d33b5532a9df" path="/var/lib/kubelet/pods/39a26fed-b9e3-4085-9f05-d33b5532a9df/volumes"
Nov 02 01:43:02.855343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.855325    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" path="/var/lib/kubelet/pods/6e1f2479-6898-45d6-8296-c68deb4b98bf/volumes"
Nov 02 01:43:02.856668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.856649    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d13070b-3087-452c-a330-c5179e3bd0cf" path="/var/lib/kubelet/pods/8d13070b-3087-452c-a330-c5179e3bd0cf/volumes"
Nov 02 01:43:02.857569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.857551    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" path="/var/lib/kubelet/pods/93c39b84-1325-4b9f-8cbf-ef64ecf31237/volumes"
Nov 02 01:43:02.858925 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.858906    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" path="/var/lib/kubelet/pods/dac1c433-8412-47ae-9794-ce298c3593bd/volumes"
Nov 02 01:43:02.966472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.966444    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"
Nov 02 01:43:02.966472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.966461    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g" event={"ID":"338501e4-000f-4aed-92d7-97cae792d292","Type":"ContainerDied","Data":"5d436749103d0b547eee39732d2867a536b47e3d79342a12d4f5d25d5ca7c935"}
Nov 02 01:43:02.966732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:02.966506    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d436749103d0b547eee39732d2867a536b47e3d79342a12d4f5d25d5ca7c935"
Nov 02 01:43:03.169019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.168986    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:43:03.217733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.217696    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a05937d-11ba-480a-8426-91ad623db8fa-kube-api-access-lpzvh\" (UniqueName: \"kubernetes.io/projected/7a05937d-11ba-480a-8426-91ad623db8fa-kube-api-access-lpzvh\") pod \"7a05937d-11ba-480a-8426-91ad623db8fa\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") "
Nov 02 01:43:03.218161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.218138    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-xks-ca\") pod \"7a05937d-11ba-480a-8426-91ad623db8fa\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") "
Nov 02 01:43:03.218464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.218440    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-openshift-ca\") pod \"7a05937d-11ba-480a-8426-91ad623db8fa\" (UID: \"7a05937d-11ba-480a-8426-91ad623db8fa\") "
Nov 02 01:43:03.218858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.218832    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-openshift-ca" pod "7a05937d-11ba-480a-8426-91ad623db8fa" (UID: "7a05937d-11ba-480a-8426-91ad623db8fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:03.219496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.219472    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:03.220525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.220496    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a05937d-11ba-480a-8426-91ad623db8fa-kube-api-access-lpzvh" pod "7a05937d-11ba-480a-8426-91ad623db8fa" (UID: "7a05937d-11ba-480a-8426-91ad623db8fa"). InnerVolumeSpecName "kube-api-access-lpzvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:03.220805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.220784    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-xks-ca" pod "7a05937d-11ba-480a-8426-91ad623db8fa" (UID: "7a05937d-11ba-480a-8426-91ad623db8fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:03.321212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.321179    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpzvh\" (UniqueName: \"kubernetes.io/projected/7a05937d-11ba-480a-8426-91ad623db8fa-kube-api-access-lpzvh\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:03.321212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.321206    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a05937d-11ba-480a-8426-91ad623db8fa-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:03.976202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.976176    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"
Nov 02 01:43:03.976202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.976187    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm" event={"ID":"7a05937d-11ba-480a-8426-91ad623db8fa","Type":"ContainerDied","Data":"8fe226ad420b5e8adcf105d45a3418a52df9c5fadd7feb8c0fed47d1075bed17"}
Nov 02 01:43:03.976684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:03.976236    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fe226ad420b5e8adcf105d45a3418a52df9c5fadd7feb8c0fed47d1075bed17"
Nov 02 01:43:05.032167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.032134    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"]
Nov 02 01:43:05.036801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.036777    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16a36ae9-988pj"]
Nov 02 01:43:05.499419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.499389    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"]
Nov 02 01:43:05.500202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500188    4851 cpu_manager.go:414] "containerMap: removing container" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy"
Nov 02 01:43:05.500246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500204    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy"
Nov 02 01:43:05.500246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500219    4851 cpu_manager.go:414] "containerMap: removing container" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy"
Nov 02 01:43:05.500246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500224    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy"
Nov 02 01:43:05.500246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500233    4851 cpu_manager.go:414] "containerMap: removing container" podUID="7a05937d-11ba-480a-8426-91ad623db8fa" containerName="revoke-keys"
Nov 02 01:43:05.500246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500239    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a05937d-11ba-480a-8426-91ad623db8fa" containerName="revoke-keys"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500262    4851 cpu_manager.go:414] "containerMap: removing container" podUID="39a26fed-b9e3-4085-9f05-d33b5532a9df" containerName="istio-proxy"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500267    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a26fed-b9e3-4085-9f05-d33b5532a9df" containerName="istio-proxy"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500278    4851 cpu_manager.go:414] "containerMap: removing container" podUID="3bcaad55-ddb6-41a5-a0dc-77715b4efe1a" containerName="revoke-keys"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500283    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bcaad55-ddb6-41a5-a0dc-77715b4efe1a" containerName="revoke-keys"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500312    4851 cpu_manager.go:414] "containerMap: removing container" podUID="338501e4-000f-4aed-92d7-97cae792d292" containerName="revoke-keys"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500317    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338501e4-000f-4aed-92d7-97cae792d292" containerName="revoke-keys"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500327    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8d13070b-3087-452c-a330-c5179e3bd0cf" containerName="istio-proxy"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500332    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d13070b-3087-452c-a330-c5179e3bd0cf" containerName="istio-proxy"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500339    4851 cpu_manager.go:414] "containerMap: removing container" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500343    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500349    4851 cpu_manager.go:414] "containerMap: removing container" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500353    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500365    4851 cpu_manager.go:414] "containerMap: removing container" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" containerName="istio-proxy"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500371    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" containerName="istio-proxy"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500381    4851 cpu_manager.go:414] "containerMap: removing container" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main"
Nov 02 01:43:05.500423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500385    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main"
Nov 02 01:43:05.501811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500500    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a26fed-b9e3-4085-9f05-d33b5532a9df" containerName="istio-proxy"
Nov 02 01:43:05.501811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500506    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bcaad55-ddb6-41a5-a0dc-77715b4efe1a" containerName="revoke-keys"
Nov 02 01:43:05.501811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500514    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="338501e4-000f-4aed-92d7-97cae792d292" containerName="revoke-keys"
Nov 02 01:43:05.501811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500521    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d13070b-3087-452c-a330-c5179e3bd0cf" containerName="istio-proxy"
Nov 02 01:43:05.501811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500529    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy"
Nov 02 01:43:05.501811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500534    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="375a3837-853f-4bb4-b078-b2116e0d80f0" containerName="istio-proxy"
Nov 02 01:43:05.501811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500541    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1aa882-4253-4421-a52c-d300b159fb5c" containerName="istio-proxy"
Nov 02 01:43:05.501811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500546    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1f2479-6898-45d6-8296-c68deb4b98bf" containerName="main"
Nov 02 01:43:05.501811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500555    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac1c433-8412-47ae-9794-ce298c3593bd" containerName="istio-proxy"
Nov 02 01:43:05.501811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.500561    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a05937d-11ba-480a-8426-91ad623db8fa" containerName="revoke-keys"
Nov 02 01:43:05.505390 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.505372    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:05.509945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.509168    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:43:05.518443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.516628    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:43:05.526465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.526439    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"]
Nov 02 01:43:05.572080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.572050    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:05.572381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.572360    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnkf\" (UniqueName: \"kubernetes.io/projected/14104127-adaf-4ae9-b6c7-2c5042cce879-kube-api-access-ljnkf\") pod \"maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:05.572504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.572484    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:05.674354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.674319    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnkf\" (UniqueName: \"kubernetes.io/projected/14104127-adaf-4ae9-b6c7-2c5042cce879-kube-api-access-ljnkf\") pod \"maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:05.674547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.674423    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:05.674613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.674551    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:05.675165 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.675142    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:05.676938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.676913    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:05.686192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.686166    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnkf\" (UniqueName: \"kubernetes.io/projected/14104127-adaf-4ae9-b6c7-2c5042cce879-kube-api-access-ljnkf\") pod \"maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:05.831322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:05.831205    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:06.132783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:06.132754    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"]
Nov 02 01:43:06.849453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:06.849370    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be843ac8-a6f4-422c-9fd9-57d4e18ef1b6" path="/var/lib/kubelet/pods/be843ac8-a6f4-422c-9fd9-57d4e18ef1b6/volumes"
Nov 02 01:43:07.023515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.023469    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz" event={"ID":"14104127-adaf-4ae9-b6c7-2c5042cce879","Type":"ContainerStarted","Data":"70f64862bc10c8934cce38c3dd2b0553378a5f242678f5c7ebe446aab5ea0b62"}
Nov 02 01:43:07.023689 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.023549    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz" event={"ID":"14104127-adaf-4ae9-b6c7-2c5042cce879","Type":"ContainerStarted","Data":"3f96f5cfac8b8b34de8f277eb4d14e394e9d9a6062c0a674b55a23fd213822f2"}
Nov 02 01:43:07.045667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.045619    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz" podStartSLOduration=2.0456066489999998 podStartE2EDuration="2.045606649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:07.042140502 +0000 UTC m=+1692.347073329" watchObservedRunningTime="2026-11-02 01:43:07.045606649 +0000 UTC m=+1692.350539469"
Nov 02 01:43:07.224785 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.224753    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"]
Nov 02 01:43:07.275647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.275610    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"]
Nov 02 01:43:07.647366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.646951    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"]
Nov 02 01:43:07.662185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.661645    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c39b84-1325-4b9f-8cbf-ef64ecf31237" containerName="istio-proxy"
Nov 02 01:43:07.688318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.685336    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"]
Nov 02 01:43:07.688318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.685507    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:07.909367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.892604    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:07.909367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.892742    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:07.909367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.892834    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k6zc\" (UniqueName: \"kubernetes.io/projected/9f75cbb6-6a11-45a6-b105-ee9ed146489b-kube-api-access-8k6zc\") pod \"maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:07.994257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.994222    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:07.994539 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.994389    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:07.994539 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.994495    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k6zc\" (UniqueName: \"kubernetes.io/projected/9f75cbb6-6a11-45a6-b105-ee9ed146489b-kube-api-access-8k6zc\") pod \"maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:07.995262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:07.995235    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:08.000701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:08.000672    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:08.008360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:08.008335    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k6zc\" (UniqueName: \"kubernetes.io/projected/9f75cbb6-6a11-45a6-b105-ee9ed146489b-kube-api-access-8k6zc\") pod \"maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:08.035652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:08.035615    4851 generic.go:356] "Generic (PLEG): container finished" podID="14104127-adaf-4ae9-b6c7-2c5042cce879" containerID="70f64862bc10c8934cce38c3dd2b0553378a5f242678f5c7ebe446aab5ea0b62" exitCode=0
Nov 02 01:43:08.035815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:08.035703    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz" event={"ID":"14104127-adaf-4ae9-b6c7-2c5042cce879","Type":"ContainerDied","Data":"70f64862bc10c8934cce38c3dd2b0553378a5f242678f5c7ebe446aab5ea0b62"}
Nov 02 01:43:08.168905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:08.168876    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:08.458429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:08.458397    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"]
Nov 02 01:43:08.458791 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:43:08.458544    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f75cbb6_6a11_45a6_b105_ee9ed146489b.slice/crio-38586411eb8549bed3cd6aa1b8ef1e4be10fd6aa1bef603c961a33e8e653bec6 WatchSource:0}: Error finding container 38586411eb8549bed3cd6aa1b8ef1e4be10fd6aa1bef603c961a33e8e653bec6: Status 404 returned error can't find the container with id 38586411eb8549bed3cd6aa1b8ef1e4be10fd6aa1bef603c961a33e8e653bec6
Nov 02 01:43:09.124897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.116805    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6" event={"ID":"9f75cbb6-6a11-45a6-b105-ee9ed146489b","Type":"ContainerStarted","Data":"f49d54d55ea1221afff9258e90a337fb2ffd8426e1c81a2f828a04460c999153"}
Nov 02 01:43:09.124897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.116869    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6" event={"ID":"9f75cbb6-6a11-45a6-b105-ee9ed146489b","Type":"ContainerStarted","Data":"38586411eb8549bed3cd6aa1b8ef1e4be10fd6aa1bef603c961a33e8e653bec6"}
Nov 02 01:43:09.316817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.316788    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:09.337775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.337659    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6" podStartSLOduration=2.337639263 podStartE2EDuration="2.337639263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:09.154685689 +0000 UTC m=+1694.459618513" watchObservedRunningTime="2026-11-02 01:43:09.337639263 +0000 UTC m=+1694.642572123"
Nov 02 01:43:09.364749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.364717    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14104127-adaf-4ae9-b6c7-2c5042cce879-kube-api-access-ljnkf\" (UniqueName: \"kubernetes.io/projected/14104127-adaf-4ae9-b6c7-2c5042cce879-kube-api-access-ljnkf\") pod \"14104127-adaf-4ae9-b6c7-2c5042cce879\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") "
Nov 02 01:43:09.364941 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.364815    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-openshift-ca\") pod \"14104127-adaf-4ae9-b6c7-2c5042cce879\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") "
Nov 02 01:43:09.364941 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.364929    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-xks-ca\") pod \"14104127-adaf-4ae9-b6c7-2c5042cce879\" (UID: \"14104127-adaf-4ae9-b6c7-2c5042cce879\") "
Nov 02 01:43:09.365208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.365175    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-openshift-ca" pod "14104127-adaf-4ae9-b6c7-2c5042cce879" (UID: "14104127-adaf-4ae9-b6c7-2c5042cce879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:09.365667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.365647    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:09.367266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.367205    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-xks-ca" pod "14104127-adaf-4ae9-b6c7-2c5042cce879" (UID: "14104127-adaf-4ae9-b6c7-2c5042cce879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:09.367661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.367638    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14104127-adaf-4ae9-b6c7-2c5042cce879-kube-api-access-ljnkf" pod "14104127-adaf-4ae9-b6c7-2c5042cce879" (UID: "14104127-adaf-4ae9-b6c7-2c5042cce879"). InnerVolumeSpecName "kube-api-access-ljnkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:09.466905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.466876    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljnkf\" (UniqueName: \"kubernetes.io/projected/14104127-adaf-4ae9-b6c7-2c5042cce879-kube-api-access-ljnkf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:09.466905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.466905    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14104127-adaf-4ae9-b6c7-2c5042cce879-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:09.641942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:09.641859    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"]
Nov 02 01:43:10.179315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:10.178724    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz" event={"ID":"14104127-adaf-4ae9-b6c7-2c5042cce879","Type":"ContainerDied","Data":"3f96f5cfac8b8b34de8f277eb4d14e394e9d9a6062c0a674b55a23fd213822f2"}
Nov 02 01:43:10.179315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:10.178795    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f96f5cfac8b8b34de8f277eb4d14e394e9d9a6062c0a674b55a23fd213822f2"
Nov 02 01:43:10.179315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:10.178864    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"
Nov 02 01:43:10.223315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:10.223183    4851 generic.go:356] "Generic (PLEG): container finished" podID="9f75cbb6-6a11-45a6-b105-ee9ed146489b" containerID="f49d54d55ea1221afff9258e90a337fb2ffd8426e1c81a2f828a04460c999153" exitCode=0
Nov 02 01:43:10.223588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:10.223539    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6" event={"ID":"9f75cbb6-6a11-45a6-b105-ee9ed146489b","Type":"ContainerDied","Data":"f49d54d55ea1221afff9258e90a337fb2ffd8426e1c81a2f828a04460c999153"}
Nov 02 01:43:11.619747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.619706    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"]
Nov 02 01:43:11.621828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.621797    4851 cpu_manager.go:414] "containerMap: removing container" podUID="14104127-adaf-4ae9-b6c7-2c5042cce879" containerName="revoke-keys"
Nov 02 01:43:11.621828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.621827    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14104127-adaf-4ae9-b6c7-2c5042cce879" containerName="revoke-keys"
Nov 02 01:43:11.622145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.622030    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="14104127-adaf-4ae9-b6c7-2c5042cce879" containerName="revoke-keys"
Nov 02 01:43:11.630574 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.630399    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"]
Nov 02 01:43:11.630574 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.630514    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:11.697962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.697937    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:11.800758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.800727    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-xks-ca\") pod \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") "
Nov 02 01:43:11.800945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.800873    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-openshift-ca\") pod \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") "
Nov 02 01:43:11.800945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.800931    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f75cbb6-6a11-45a6-b105-ee9ed146489b-kube-api-access-8k6zc\" (UniqueName: \"kubernetes.io/projected/9f75cbb6-6a11-45a6-b105-ee9ed146489b-kube-api-access-8k6zc\") pod \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\" (UID: \"9f75cbb6-6a11-45a6-b105-ee9ed146489b\") "
Nov 02 01:43:11.801355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.801329    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-openshift-ca" pod "9f75cbb6-6a11-45a6-b105-ee9ed146489b" (UID: "9f75cbb6-6a11-45a6-b105-ee9ed146489b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:11.801475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.801370    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:11.801686 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.801653    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcmcb\" (UniqueName: \"kubernetes.io/projected/142f9451-74dc-4d9f-9210-74dd3d893ec8-kube-api-access-xcmcb\") pod \"maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:11.801799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.801779    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:11.801905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.801892    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:11.803283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.803255    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-xks-ca" pod "9f75cbb6-6a11-45a6-b105-ee9ed146489b" (UID: "9f75cbb6-6a11-45a6-b105-ee9ed146489b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:11.803283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.803317    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f75cbb6-6a11-45a6-b105-ee9ed146489b-kube-api-access-8k6zc" pod "9f75cbb6-6a11-45a6-b105-ee9ed146489b" (UID: "9f75cbb6-6a11-45a6-b105-ee9ed146489b"). InnerVolumeSpecName "kube-api-access-8k6zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:11.903359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.903204    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:11.903551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.903413    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:11.903640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.903622    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcmcb\" (UniqueName: \"kubernetes.io/projected/142f9451-74dc-4d9f-9210-74dd3d893ec8-kube-api-access-xcmcb\") pod \"maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:11.903725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.903712    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f75cbb6-6a11-45a6-b105-ee9ed146489b-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:11.903787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.903726    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k6zc\" (UniqueName: \"kubernetes.io/projected/9f75cbb6-6a11-45a6-b105-ee9ed146489b-kube-api-access-8k6zc\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:11.904072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.904047    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:11.906174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.906151    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:11.915938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.915907    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcmcb\" (UniqueName: \"kubernetes.io/projected/142f9451-74dc-4d9f-9210-74dd3d893ec8-kube-api-access-xcmcb\") pod \"maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:11.949334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:11.949264    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:12.244769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:12.225115    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p" podUID="bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd" containerName="payload-processing" containerID="cri-o://491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06" gracePeriod=25
Nov 02 01:43:12.260830 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:12.260402    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6" event={"ID":"9f75cbb6-6a11-45a6-b105-ee9ed146489b","Type":"ContainerDied","Data":"38586411eb8549bed3cd6aa1b8ef1e4be10fd6aa1bef603c961a33e8e653bec6"}
Nov 02 01:43:12.260830 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:12.260522    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38586411eb8549bed3cd6aa1b8ef1e4be10fd6aa1bef603c961a33e8e653bec6"
Nov 02 01:43:12.260830 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:12.260799    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"
Nov 02 01:43:12.268264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:12.268225    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"]
Nov 02 01:43:12.278021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:12.277959    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m" podUID="ce77099f-2ea7-4032-ae62-9e089de80c22" containerName="payload-pre-processing" containerID="cri-o://1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9" gracePeriod=25
Nov 02 01:43:13.271160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:13.271122    4851 generic.go:356] "Generic (PLEG): container finished" podID="142f9451-74dc-4d9f-9210-74dd3d893ec8" containerID="66658adc13a26d5c359172f6782656988e7617966c238f2bcea78ddf66a7d6f9" exitCode=0
Nov 02 01:43:13.271601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:13.271202    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp" event={"ID":"142f9451-74dc-4d9f-9210-74dd3d893ec8","Type":"ContainerDied","Data":"66658adc13a26d5c359172f6782656988e7617966c238f2bcea78ddf66a7d6f9"}
Nov 02 01:43:13.271601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:13.271258    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp" event={"ID":"142f9451-74dc-4d9f-9210-74dd3d893ec8","Type":"ContainerStarted","Data":"91c5b2ee6a12e68c113e8c9726049b6af4c7d99bafbc733c869ded5a84107533"}
Nov 02 01:43:14.560011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.559981    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:14.642940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.642829    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s" podUID="9a157216-0fe8-496b-811e-35ee545e3e31" containerName="payload-processing" containerID="cri-o://c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376" gracePeriod=25
Nov 02 01:43:14.667555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.667524    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142f9451-74dc-4d9f-9210-74dd3d893ec8-kube-api-access-xcmcb\" (UniqueName: \"kubernetes.io/projected/142f9451-74dc-4d9f-9210-74dd3d893ec8-kube-api-access-xcmcb\") pod \"142f9451-74dc-4d9f-9210-74dd3d893ec8\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") "
Nov 02 01:43:14.667696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.667652    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-xks-ca\") pod \"142f9451-74dc-4d9f-9210-74dd3d893ec8\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") "
Nov 02 01:43:14.667759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.667695    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-openshift-ca\") pod \"142f9451-74dc-4d9f-9210-74dd3d893ec8\" (UID: \"142f9451-74dc-4d9f-9210-74dd3d893ec8\") "
Nov 02 01:43:14.668073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.668049    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-openshift-ca" pod "142f9451-74dc-4d9f-9210-74dd3d893ec8" (UID: "142f9451-74dc-4d9f-9210-74dd3d893ec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:14.668451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.668431    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:14.669868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.669843    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142f9451-74dc-4d9f-9210-74dd3d893ec8-kube-api-access-xcmcb" pod "142f9451-74dc-4d9f-9210-74dd3d893ec8" (UID: "142f9451-74dc-4d9f-9210-74dd3d893ec8"). InnerVolumeSpecName "kube-api-access-xcmcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:14.669974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.669961    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-xks-ca" pod "142f9451-74dc-4d9f-9210-74dd3d893ec8" (UID: "142f9451-74dc-4d9f-9210-74dd3d893ec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:14.769827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.769798    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcmcb\" (UniqueName: \"kubernetes.io/projected/142f9451-74dc-4d9f-9210-74dd3d893ec8-kube-api-access-xcmcb\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:14.769827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.769826    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142f9451-74dc-4d9f-9210-74dd3d893ec8-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:14.952123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:14.952088    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:43:15.102727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.093897    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a157216-0fe8-496b-811e-35ee545e3e31-kube-api-access-cvxf8\" (UniqueName: \"kubernetes.io/projected/9a157216-0fe8-496b-811e-35ee545e3e31-kube-api-access-cvxf8\") pod \"9a157216-0fe8-496b-811e-35ee545e3e31\" (UID: \"9a157216-0fe8-496b-811e-35ee545e3e31\") "
Nov 02 01:43:15.102727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.094251    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a157216-0fe8-496b-811e-35ee545e3e31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a157216-0fe8-496b-811e-35ee545e3e31-plugins-config-volume\") pod \"9a157216-0fe8-496b-811e-35ee545e3e31\" (UID: \"9a157216-0fe8-496b-811e-35ee545e3e31\") "
Nov 02 01:43:15.114408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.109345    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a157216-0fe8-496b-811e-35ee545e3e31-kube-api-access-cvxf8" pod "9a157216-0fe8-496b-811e-35ee545e3e31" (UID: "9a157216-0fe8-496b-811e-35ee545e3e31"). InnerVolumeSpecName "kube-api-access-cvxf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:15.131314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.128769    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a157216-0fe8-496b-811e-35ee545e3e31-plugins-config-volume" pod "9a157216-0fe8-496b-811e-35ee545e3e31" (UID: "9a157216-0fe8-496b-811e-35ee545e3e31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:15.196714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.196635    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvxf8\" (UniqueName: \"kubernetes.io/projected/9a157216-0fe8-496b-811e-35ee545e3e31-kube-api-access-cvxf8\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:15.196714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.196664    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a157216-0fe8-496b-811e-35ee545e3e31-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:15.325813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.325781    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"
Nov 02 01:43:15.326006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.325814    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp" event={"ID":"142f9451-74dc-4d9f-9210-74dd3d893ec8","Type":"ContainerDied","Data":"91c5b2ee6a12e68c113e8c9726049b6af4c7d99bafbc733c869ded5a84107533"}
Nov 02 01:43:15.326006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.325861    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91c5b2ee6a12e68c113e8c9726049b6af4c7d99bafbc733c869ded5a84107533"
Nov 02 01:43:15.327497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.327472    4851 generic.go:356] "Generic (PLEG): container finished" podID="9a157216-0fe8-496b-811e-35ee545e3e31" containerID="c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376" exitCode=0
Nov 02 01:43:15.327622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.327518    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"
Nov 02 01:43:15.327622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.327527    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s" event={"ID":"9a157216-0fe8-496b-811e-35ee545e3e31","Type":"ContainerDied","Data":"c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376"}
Nov 02 01:43:15.327622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.327573    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s" event={"ID":"9a157216-0fe8-496b-811e-35ee545e3e31","Type":"ContainerDied","Data":"920a491edf667a0ae7c5e270fff2dfbbebc6c3f2e95c58427f85d24b302beb2e"}
Nov 02 01:43:15.327622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.327604    4851 scope.go:122] "RemoveContainer" containerID="c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376"
Nov 02 01:43:15.346498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.346462    4851 scope.go:122] "RemoveContainer" containerID="c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376"
Nov 02 01:43:15.346890 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:15.346867    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376\": container with ID starting with c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376 not found: ID does not exist" containerID="c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376"
Nov 02 01:43:15.347003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.346898    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376"} err="failed to get container status \"c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376\": rpc error: code = NotFound desc = could not find container \"c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376\": container with ID starting with c258df457341f5f67079a1b404d59f7c73347862e126efaec1d90aecf1d63376 not found: ID does not exist"
Nov 02 01:43:15.366551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.366518    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"]
Nov 02 01:43:15.371719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.371681    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-addad9-5d69c9f8c9-kc76s"]
Nov 02 01:43:15.637832 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.637797    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"]
Nov 02 01:43:15.655476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:15.655438    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"]
Nov 02 01:43:16.541731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:16.541696    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"]
Nov 02 01:43:16.541971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:16.541939    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn" podUID="4bc9fe1d-e29c-4d82-919a-58d4d6e9364e" containerName="maas-api" containerID="cri-o://9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905" gracePeriod=30
Nov 02 01:43:16.908368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:16.907134    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a157216-0fe8-496b-811e-35ee545e3e31" path="/var/lib/kubelet/pods/9a157216-0fe8-496b-811e-35ee545e3e31/volumes"
Nov 02 01:43:17.053627 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.053577    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"]
Nov 02 01:43:17.061722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.061692    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-14b63f-2mjzz"]
Nov 02 01:43:17.110757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.110735    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:43:17.316974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.306965    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-openshift-service-ca\") pod \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") "
Nov 02 01:43:17.316974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.307204    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-kube-api-access-dd9sp\" (UniqueName: \"kubernetes.io/projected/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-kube-api-access-dd9sp\") pod \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") "
Nov 02 01:43:17.316974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.307267    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-maas-api-tls\") pod \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") "
Nov 02 01:43:17.316974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.310140    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-openshift-service-ca" pod "4bc9fe1d-e29c-4d82-919a-58d4d6e9364e" (UID: "4bc9fe1d-e29c-4d82-919a-58d4d6e9364e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:17.316974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.313635    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-metrics-certs\") pod \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\" (UID: \"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e\") "
Nov 02 01:43:17.316974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.314816    4851 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-openshift-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:17.320513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.320479    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-kube-api-access-dd9sp" pod "4bc9fe1d-e29c-4d82-919a-58d4d6e9364e" (UID: "4bc9fe1d-e29c-4d82-919a-58d4d6e9364e"). InnerVolumeSpecName "kube-api-access-dd9sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:17.320897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.320872    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-metrics-certs" pod "4bc9fe1d-e29c-4d82-919a-58d4d6e9364e" (UID: "4bc9fe1d-e29c-4d82-919a-58d4d6e9364e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:17.321036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.321011    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-maas-api-tls" pod "4bc9fe1d-e29c-4d82-919a-58d4d6e9364e" (UID: "4bc9fe1d-e29c-4d82-919a-58d4d6e9364e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:17.367770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.367739    4851 generic.go:356] "Generic (PLEG): container finished" podID="4bc9fe1d-e29c-4d82-919a-58d4d6e9364e" containerID="9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905" exitCode=0
Nov 02 01:43:17.367895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.367787    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"
Nov 02 01:43:17.367895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.367826    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn" event={"ID":"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e","Type":"ContainerDied","Data":"9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905"}
Nov 02 01:43:17.367895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.367884    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn" event={"ID":"4bc9fe1d-e29c-4d82-919a-58d4d6e9364e","Type":"ContainerDied","Data":"72c9dfb25ab8a3642787c437dcccd9e025d223a45f3d6eea039f854ce276642b"}
Nov 02 01:43:17.368012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.367913    4851 scope.go:122] "RemoveContainer" containerID="9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905"
Nov 02 01:43:17.379617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.379600    4851 scope.go:122] "RemoveContainer" containerID="9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905"
Nov 02 01:43:17.379973 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:17.379955    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905\": container with ID starting with 9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905 not found: ID does not exist" containerID="9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905"
Nov 02 01:43:17.380086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.379982    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905"} err="failed to get container status \"9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905\": rpc error: code = NotFound desc = could not find container \"9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905\": container with ID starting with 9933d90f97de8884b2f8a2df561a9b1a9c98e08bb2d2038d51ae507d02782905 not found: ID does not exist"
Nov 02 01:43:17.394437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.394402    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"]
Nov 02 01:43:17.398977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.398956    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-14b63f-577d894fd7-7j5nn"]
Nov 02 01:43:17.416882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.416856    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd9sp\" (UniqueName: \"kubernetes.io/projected/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-kube-api-access-dd9sp\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:17.416996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.416884    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-maas-api-tls\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:17.416996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:17.416895    4851 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e-metrics-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:17.658110 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:17.658031    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:43:18.744463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:18.744428    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"]
Nov 02 01:43:18.760854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:18.760825    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e412d5-smbd6"]
Nov 02 01:43:18.862781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:18.859767    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14104127-adaf-4ae9-b6c7-2c5042cce879" path="/var/lib/kubelet/pods/14104127-adaf-4ae9-b6c7-2c5042cce879/volumes"
Nov 02 01:43:18.862781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:18.860722    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc9fe1d-e29c-4d82-919a-58d4d6e9364e" path="/var/lib/kubelet/pods/4bc9fe1d-e29c-4d82-919a-58d4d6e9364e/volumes"
Nov 02 01:43:18.862781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:18.861469    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f75cbb6-6a11-45a6-b105-ee9ed146489b" path="/var/lib/kubelet/pods/9f75cbb6-6a11-45a6-b105-ee9ed146489b/volumes"
Nov 02 01:43:20.293984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:20.293947    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m" podUID="ce77099f-2ea7-4032-ae62-9e089de80c22" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:43:20.293984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:20.293961    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p" podUID="bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:43:20.638244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:20.638148    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7" podUID="9241dd32-519f-47a0-b614-2211204c6e8b" containerName="payload-processing" containerID="cri-o://440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc" gracePeriod=25
Nov 02 01:43:20.669316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:20.663254    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn" podUID="e7f9e8df-23b6-478d-b95b-00a6989afed7" containerName="payload-pre-processing" containerID="cri-o://4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8" gracePeriod=25
Nov 02 01:43:21.025470 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.025082    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:43:21.056732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.056699    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:43:21.183885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.183841    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9241dd32-519f-47a0-b614-2211204c6e8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9241dd32-519f-47a0-b614-2211204c6e8b-plugins-config-volume\") pod \"9241dd32-519f-47a0-b614-2211204c6e8b\" (UID: \"9241dd32-519f-47a0-b614-2211204c6e8b\") "
Nov 02 01:43:21.184166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.184142    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f9e8df-23b6-478d-b95b-00a6989afed7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f9e8df-23b6-478d-b95b-00a6989afed7-plugins-config-volume\") pod \"e7f9e8df-23b6-478d-b95b-00a6989afed7\" (UID: \"e7f9e8df-23b6-478d-b95b-00a6989afed7\") "
Nov 02 01:43:21.185018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.184995    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f9e8df-23b6-478d-b95b-00a6989afed7-kube-api-access-qq8zx\" (UniqueName: \"kubernetes.io/projected/e7f9e8df-23b6-478d-b95b-00a6989afed7-kube-api-access-qq8zx\") pod \"e7f9e8df-23b6-478d-b95b-00a6989afed7\" (UID: \"e7f9e8df-23b6-478d-b95b-00a6989afed7\") "
Nov 02 01:43:21.185621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.185600    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9241dd32-519f-47a0-b614-2211204c6e8b-kube-api-access-nw9gv\" (UniqueName: \"kubernetes.io/projected/9241dd32-519f-47a0-b614-2211204c6e8b-kube-api-access-nw9gv\") pod \"9241dd32-519f-47a0-b614-2211204c6e8b\" (UID: \"9241dd32-519f-47a0-b614-2211204c6e8b\") "
Nov 02 01:43:21.189575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.184528    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9241dd32-519f-47a0-b614-2211204c6e8b-plugins-config-volume" pod "9241dd32-519f-47a0-b614-2211204c6e8b" (UID: "9241dd32-519f-47a0-b614-2211204c6e8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:21.189924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.184655    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f9e8df-23b6-478d-b95b-00a6989afed7-plugins-config-volume" pod "e7f9e8df-23b6-478d-b95b-00a6989afed7" (UID: "e7f9e8df-23b6-478d-b95b-00a6989afed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:21.190988 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.190968    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9241dd32-519f-47a0-b614-2211204c6e8b-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:21.192772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.191125    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f9e8df-23b6-478d-b95b-00a6989afed7-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:21.203743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.203712    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f9e8df-23b6-478d-b95b-00a6989afed7-kube-api-access-qq8zx" pod "e7f9e8df-23b6-478d-b95b-00a6989afed7" (UID: "e7f9e8df-23b6-478d-b95b-00a6989afed7"). InnerVolumeSpecName "kube-api-access-qq8zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:21.205232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.205205    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9241dd32-519f-47a0-b614-2211204c6e8b-kube-api-access-nw9gv" pod "9241dd32-519f-47a0-b614-2211204c6e8b" (UID: "9241dd32-519f-47a0-b614-2211204c6e8b"). InnerVolumeSpecName "kube-api-access-nw9gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:21.292918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.292836    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq8zx\" (UniqueName: \"kubernetes.io/projected/e7f9e8df-23b6-478d-b95b-00a6989afed7-kube-api-access-qq8zx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:21.292918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.292864    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9gv\" (UniqueName: \"kubernetes.io/projected/9241dd32-519f-47a0-b614-2211204c6e8b-kube-api-access-nw9gv\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:21.554369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.550483    4851 generic.go:356] "Generic (PLEG): container finished" podID="e7f9e8df-23b6-478d-b95b-00a6989afed7" containerID="4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8" exitCode=0
Nov 02 01:43:21.554369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.550792    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn" event={"ID":"e7f9e8df-23b6-478d-b95b-00a6989afed7","Type":"ContainerDied","Data":"4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8"}
Nov 02 01:43:21.554369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.550848    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn" event={"ID":"e7f9e8df-23b6-478d-b95b-00a6989afed7","Type":"ContainerDied","Data":"de48d6d16fef3ed99373e91b5ed10eac9f78f8f0ab1bd96e3315b5d48ea77fee"}
Nov 02 01:43:21.569316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.566415    4851 scope.go:122] "RemoveContainer" containerID="4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8"
Nov 02 01:43:21.569316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.567384    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"
Nov 02 01:43:21.621318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.620320    4851 generic.go:356] "Generic (PLEG): container finished" podID="9241dd32-519f-47a0-b614-2211204c6e8b" containerID="440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc" exitCode=0
Nov 02 01:43:21.621318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.620434    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7" event={"ID":"9241dd32-519f-47a0-b614-2211204c6e8b","Type":"ContainerDied","Data":"440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc"}
Nov 02 01:43:21.621318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.620489    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7" event={"ID":"9241dd32-519f-47a0-b614-2211204c6e8b","Type":"ContainerDied","Data":"dd9393d79e91d769814ed7ec43315ffd6528178d575239c53212382fb29bba7b"}
Nov 02 01:43:21.621318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.620574    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"
Nov 02 01:43:21.851926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.835890    4851 scope.go:122] "RemoveContainer" containerID="4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8"
Nov 02 01:43:21.851926 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:21.837395    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8\": container with ID starting with 4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8 not found: ID does not exist" containerID="4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8"
Nov 02 01:43:21.851926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.837455    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8"} err="failed to get container status \"4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8\": rpc error: code = NotFound desc = could not find container \"4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8\": container with ID starting with 4b53d6b832147f79453105cb3dbc5d5deba612141c9c6acbc1dae1c8f03e5eb8 not found: ID does not exist"
Nov 02 01:43:21.851926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.837485    4851 scope.go:122] "RemoveContainer" containerID="440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc"
Nov 02 01:43:21.925553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.920208    4851 scope.go:122] "RemoveContainer" containerID="440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc"
Nov 02 01:43:21.925553 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:21.922719    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc\": container with ID starting with 440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc not found: ID does not exist" containerID="440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc"
Nov 02 01:43:21.925553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.922796    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc"} err="failed to get container status \"440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc\": rpc error: code = NotFound desc = could not find container \"440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc\": container with ID starting with 440dcdf00c65f302d1c52cd6ee1382c7a2122d67d8980f9f9fbd7563a83213fc not found: ID does not exist"
Nov 02 01:43:21.934542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.934505    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"]
Nov 02 01:43:21.939817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.939786    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-14b63f-747b49b599-p95nn"]
Nov 02 01:43:21.945599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.945571    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"]
Nov 02 01:43:21.950368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:21.950340    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-14b63f-66bb67cb67-96cd7"]
Nov 02 01:43:22.047034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:22.046998    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"]
Nov 02 01:43:22.053768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:22.053737    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-addad9-7h9pm"]
Nov 02 01:43:22.851370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:22.851338    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a05937d-11ba-480a-8426-91ad623db8fa" path="/var/lib/kubelet/pods/7a05937d-11ba-480a-8426-91ad623db8fa/volumes"
Nov 02 01:43:22.851882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:22.851838    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9241dd32-519f-47a0-b614-2211204c6e8b" path="/var/lib/kubelet/pods/9241dd32-519f-47a0-b614-2211204c6e8b/volumes"
Nov 02 01:43:22.852339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:22.852281    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f9e8df-23b6-478d-b95b-00a6989afed7" path="/var/lib/kubelet/pods/e7f9e8df-23b6-478d-b95b-00a6989afed7/volumes"
Nov 02 01:43:25.143621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.143315    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"]
Nov 02 01:43:25.145962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.145926    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9241dd32-519f-47a0-b614-2211204c6e8b" containerName="payload-processing"
Nov 02 01:43:25.145962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.145958    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9241dd32-519f-47a0-b614-2211204c6e8b" containerName="payload-processing"
Nov 02 01:43:25.146149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146014    4851 cpu_manager.go:414] "containerMap: removing container" podUID="e7f9e8df-23b6-478d-b95b-00a6989afed7" containerName="payload-pre-processing"
Nov 02 01:43:25.146149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146023    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f9e8df-23b6-478d-b95b-00a6989afed7" containerName="payload-pre-processing"
Nov 02 01:43:25.146149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146040    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9f75cbb6-6a11-45a6-b105-ee9ed146489b" containerName="revoke-keys"
Nov 02 01:43:25.146149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146048    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f75cbb6-6a11-45a6-b105-ee9ed146489b" containerName="revoke-keys"
Nov 02 01:43:25.146149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146063    4851 cpu_manager.go:414] "containerMap: removing container" podUID="4bc9fe1d-e29c-4d82-919a-58d4d6e9364e" containerName="maas-api"
Nov 02 01:43:25.146149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146071    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc9fe1d-e29c-4d82-919a-58d4d6e9364e" containerName="maas-api"
Nov 02 01:43:25.146149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146088    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9a157216-0fe8-496b-811e-35ee545e3e31" containerName="payload-processing"
Nov 02 01:43:25.146149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146095    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a157216-0fe8-496b-811e-35ee545e3e31" containerName="payload-processing"
Nov 02 01:43:25.146149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146117    4851 cpu_manager.go:414] "containerMap: removing container" podUID="142f9451-74dc-4d9f-9210-74dd3d893ec8" containerName="revoke-keys"
Nov 02 01:43:25.146149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146124    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142f9451-74dc-4d9f-9210-74dd3d893ec8" containerName="revoke-keys"
Nov 02 01:43:25.146639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146330    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9241dd32-519f-47a0-b614-2211204c6e8b" containerName="payload-processing"
Nov 02 01:43:25.146639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146352    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f9e8df-23b6-478d-b95b-00a6989afed7" containerName="payload-pre-processing"
Nov 02 01:43:25.146639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146363    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f75cbb6-6a11-45a6-b105-ee9ed146489b" containerName="revoke-keys"
Nov 02 01:43:25.146639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146371    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc9fe1d-e29c-4d82-919a-58d4d6e9364e" containerName="maas-api"
Nov 02 01:43:25.146639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146382    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a157216-0fe8-496b-811e-35ee545e3e31" containerName="payload-processing"
Nov 02 01:43:25.146639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.146392    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="142f9451-74dc-4d9f-9210-74dd3d893ec8" containerName="revoke-keys"
Nov 02 01:43:25.154400 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.154368    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.155679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.155201    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"]
Nov 02 01:43:25.159069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.159038    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:43:25.159069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.159044    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:43:25.205820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.205778    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:43:25.206023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.205926    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:43:25.297893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.297848    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.298647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.298617    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.299272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.299240    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7k5\" (UniqueName: \"kubernetes.io/projected/f52e3ca9-d2cc-4607-b160-1e963871853f-kube-api-access-5z7k5\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.400945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.400846    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.401134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.400960    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7k5\" (UniqueName: \"kubernetes.io/projected/f52e3ca9-d2cc-4607-b160-1e963871853f-kube-api-access-5z7k5\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.401204 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.401167    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.401556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.401535    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.403732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.403707    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.413520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.413494    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7k5\" (UniqueName: \"kubernetes.io/projected/f52e3ca9-d2cc-4607-b160-1e963871853f-kube-api-access-5z7k5\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.474980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.474949    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:25.761173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.761139    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"]
Nov 02 01:43:25.761372 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:43:25.761350    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52e3ca9_d2cc_4607_b160_1e963871853f.slice/crio-103c4edeb5bf03a698912ddbb6fa986d949609c13f37e5f6d795f6abd8658f83 WatchSource:0}: Error finding container 103c4edeb5bf03a698912ddbb6fa986d949609c13f37e5f6d795f6abd8658f83: Status 404 returned error can't find the container with id 103c4edeb5bf03a698912ddbb6fa986d949609c13f37e5f6d795f6abd8658f83
Nov 02 01:43:25.836001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:25.835966    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w" event={"ID":"f52e3ca9-d2cc-4607-b160-1e963871853f","Type":"ContainerStarted","Data":"103c4edeb5bf03a698912ddbb6fa986d949609c13f37e5f6d795f6abd8658f83"}
Nov 02 01:43:27.010539 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.010475    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w" podStartSLOduration=2.010455422 podStartE2EDuration="2.010455422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:26.966070667 +0000 UTC m=+1712.271003495" watchObservedRunningTime="2026-11-02 01:43:27.010455422 +0000 UTC m=+1712.315388248"
Nov 02 01:43:27.014469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.014426    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w" event={"ID":"f52e3ca9-d2cc-4607-b160-1e963871853f","Type":"ContainerStarted","Data":"821f3e968476e466723c96e060839604f4c81b4aefc8d79af189c127c8ba7e18"}
Nov 02 01:43:27.382378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.380334    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"]
Nov 02 01:43:27.401314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.399109    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.426947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.425937    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"]
Nov 02 01:43:27.529854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.529814    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.530069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.529942    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cn57\" (UniqueName: \"kubernetes.io/projected/d9c36349-4646-4bcd-9b40-ede290c97edc-kube-api-access-2cn57\") pod \"maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.530117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.530080    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.631603 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.631566    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.631783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.631637    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cn57\" (UniqueName: \"kubernetes.io/projected/d9c36349-4646-4bcd-9b40-ede290c97edc-kube-api-access-2cn57\") pod \"maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.631783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.631689    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.632255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.632228    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.634227 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.634159    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.645146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.645110    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cn57\" (UniqueName: \"kubernetes.io/projected/d9c36349-4646-4bcd-9b40-ede290c97edc-kube-api-access-2cn57\") pod \"maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.749365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.749274    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:27.948896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.948855    4851 generic.go:356] "Generic (PLEG): container finished" podID="f52e3ca9-d2cc-4607-b160-1e963871853f" containerID="821f3e968476e466723c96e060839604f4c81b4aefc8d79af189c127c8ba7e18" exitCode=0
Nov 02 01:43:27.949122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:27.948911    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w" event={"ID":"f52e3ca9-d2cc-4607-b160-1e963871853f","Type":"ContainerDied","Data":"821f3e968476e466723c96e060839604f4c81b4aefc8d79af189c127c8ba7e18"}
Nov 02 01:43:28.041168 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:43:28.041136    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c36349_4646_4bcd_9b40_ede290c97edc.slice/crio-84400a27cf5b14be292378189aff0b961ff128b67e8bfde82d1785ce35f71959 WatchSource:0}: Error finding container 84400a27cf5b14be292378189aff0b961ff128b67e8bfde82d1785ce35f71959: Status 404 returned error can't find the container with id 84400a27cf5b14be292378189aff0b961ff128b67e8bfde82d1785ce35f71959
Nov 02 01:43:28.041673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:28.041644    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"]
Nov 02 01:43:28.961945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:28.961905    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8" event={"ID":"d9c36349-4646-4bcd-9b40-ede290c97edc","Type":"ContainerStarted","Data":"58aab98182e964349ad1a30c68d6d80218405f84683d5a26efebfffb956db4aa"}
Nov 02 01:43:28.962158 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:28.961967    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8" event={"ID":"d9c36349-4646-4bcd-9b40-ede290c97edc","Type":"ContainerStarted","Data":"84400a27cf5b14be292378189aff0b961ff128b67e8bfde82d1785ce35f71959"}
Nov 02 01:43:28.982083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:28.982022    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8" podStartSLOduration=1.982008513 podStartE2EDuration="1.982008513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:28.980593832 +0000 UTC m=+1714.285526659" watchObservedRunningTime="2026-11-02 01:43:28.982008513 +0000 UTC m=+1714.286941339"
Nov 02 01:43:29.541547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.541519    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:29.541910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.541584    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"]
Nov 02 01:43:29.546267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.546238    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0bbeda-4sscs"]
Nov 02 01:43:29.726312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.719133    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-xks-ca\") pod \"f52e3ca9-d2cc-4607-b160-1e963871853f\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") "
Nov 02 01:43:29.726312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.719217    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-openshift-ca\") pod \"f52e3ca9-d2cc-4607-b160-1e963871853f\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") "
Nov 02 01:43:29.726312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.719368    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52e3ca9-d2cc-4607-b160-1e963871853f-kube-api-access-5z7k5\" (UniqueName: \"kubernetes.io/projected/f52e3ca9-d2cc-4607-b160-1e963871853f-kube-api-access-5z7k5\") pod \"f52e3ca9-d2cc-4607-b160-1e963871853f\" (UID: \"f52e3ca9-d2cc-4607-b160-1e963871853f\") "
Nov 02 01:43:29.726312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.721617    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-openshift-ca" pod "f52e3ca9-d2cc-4607-b160-1e963871853f" (UID: "f52e3ca9-d2cc-4607-b160-1e963871853f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:29.735522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.731592    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52e3ca9-d2cc-4607-b160-1e963871853f-kube-api-access-5z7k5" pod "f52e3ca9-d2cc-4607-b160-1e963871853f" (UID: "f52e3ca9-d2cc-4607-b160-1e963871853f"). InnerVolumeSpecName "kube-api-access-5z7k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:29.750451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.749375    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-xks-ca" pod "f52e3ca9-d2cc-4607-b160-1e963871853f" (UID: "f52e3ca9-d2cc-4607-b160-1e963871853f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:29.822119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.822089    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:29.822119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.822119    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f52e3ca9-d2cc-4607-b160-1e963871853f-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:29.822352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.822132    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z7k5\" (UniqueName: \"kubernetes.io/projected/f52e3ca9-d2cc-4607-b160-1e963871853f-kube-api-access-5z7k5\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:29.972577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.972546    4851 generic.go:356] "Generic (PLEG): container finished" podID="d9c36349-4646-4bcd-9b40-ede290c97edc" containerID="58aab98182e964349ad1a30c68d6d80218405f84683d5a26efebfffb956db4aa" exitCode=0
Nov 02 01:43:29.972757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.972625    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8" event={"ID":"d9c36349-4646-4bcd-9b40-ede290c97edc","Type":"ContainerDied","Data":"58aab98182e964349ad1a30c68d6d80218405f84683d5a26efebfffb956db4aa"}
Nov 02 01:43:29.974045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.974027    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"
Nov 02 01:43:29.974166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.974072    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w" event={"ID":"f52e3ca9-d2cc-4607-b160-1e963871853f","Type":"ContainerDied","Data":"103c4edeb5bf03a698912ddbb6fa986d949609c13f37e5f6d795f6abd8658f83"}
Nov 02 01:43:29.974166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:29.974110    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="103c4edeb5bf03a698912ddbb6fa986d949609c13f37e5f6d795f6abd8658f83"
Nov 02 01:43:30.232796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.232761    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"]
Nov 02 01:43:30.233121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.233015    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy" containerID="cri-o://36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f" gracePeriod=30
Nov 02 01:43:30.295257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.295212    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p" podUID="bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:43:30.295460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.295396    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m" podUID="ce77099f-2ea7-4032-ae62-9e089de80c22" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:43:30.777105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.777071    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"]
Nov 02 01:43:30.777904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.777888    4851 cpu_manager.go:414] "containerMap: removing container" podUID="f52e3ca9-d2cc-4607-b160-1e963871853f" containerName="revoke-keys"
Nov 02 01:43:30.777957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.777905    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52e3ca9-d2cc-4607-b160-1e963871853f" containerName="revoke-keys"
Nov 02 01:43:30.778075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.778065    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52e3ca9-d2cc-4607-b160-1e963871853f" containerName="revoke-keys"
Nov 02 01:43:30.782044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.782023    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:30.791095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.791067    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"]
Nov 02 01:43:30.797023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.796991    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:30.797203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.797181    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:30.798065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.797449    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcscf\" (UniqueName: \"kubernetes.io/projected/e6d652d1-d6ae-4e0f-b41f-f7204737b226-kube-api-access-kcscf\") pod \"maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:30.852663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.852610    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bcaad55-ddb6-41a5-a0dc-77715b4efe1a" path="/var/lib/kubelet/pods/3bcaad55-ddb6-41a5-a0dc-77715b4efe1a/volumes"
Nov 02 01:43:30.899745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.899704    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:30.899963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.899826    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:30.900036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.899999    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcscf\" (UniqueName: \"kubernetes.io/projected/e6d652d1-d6ae-4e0f-b41f-f7204737b226-kube-api-access-kcscf\") pod \"maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:30.900641 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.900556    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:30.902344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.902316    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:30.912036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:30.912015    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcscf\" (UniqueName: \"kubernetes.io/projected/e6d652d1-d6ae-4e0f-b41f-f7204737b226-kube-api-access-kcscf\") pod \"maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:31.106103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.106069    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:31.187003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.186969    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:31.273413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.246152    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-openshift-ca\") pod \"d9c36349-4646-4bcd-9b40-ede290c97edc\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") "
Nov 02 01:43:31.273413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.246270    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-xks-ca\") pod \"d9c36349-4646-4bcd-9b40-ede290c97edc\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") "
Nov 02 01:43:31.273413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.246369    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c36349-4646-4bcd-9b40-ede290c97edc-kube-api-access-2cn57\" (UniqueName: \"kubernetes.io/projected/d9c36349-4646-4bcd-9b40-ede290c97edc-kube-api-access-2cn57\") pod \"d9c36349-4646-4bcd-9b40-ede290c97edc\" (UID: \"d9c36349-4646-4bcd-9b40-ede290c97edc\") "
Nov 02 01:43:31.294987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.294928    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-openshift-ca" pod "d9c36349-4646-4bcd-9b40-ede290c97edc" (UID: "d9c36349-4646-4bcd-9b40-ede290c97edc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:31.298745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.297459    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-xks-ca" pod "d9c36349-4646-4bcd-9b40-ede290c97edc" (UID: "d9c36349-4646-4bcd-9b40-ede290c97edc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:31.352270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.352227    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:31.366348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.359040    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c36349-4646-4bcd-9b40-ede290c97edc-kube-api-access-2cn57" pod "d9c36349-4646-4bcd-9b40-ede290c97edc" (UID: "d9c36349-4646-4bcd-9b40-ede290c97edc"). InnerVolumeSpecName "kube-api-access-2cn57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:31.366592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.366278    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c36349-4646-4bcd-9b40-ede290c97edc-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:31.468204 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.468156    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cn57\" (UniqueName: \"kubernetes.io/projected/d9c36349-4646-4bcd-9b40-ede290c97edc-kube-api-access-2cn57\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:31.522927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.522888    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"]
Nov 02 01:43:31.524181 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:43:31.524149    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d652d1_d6ae_4e0f_b41f_f7204737b226.slice/crio-e2a1262577c5843a16ce29d02227e1eb8b51e6cfb4114edb64c9a2015add5ba8 WatchSource:0}: Error finding container e2a1262577c5843a16ce29d02227e1eb8b51e6cfb4114edb64c9a2015add5ba8: Status 404 returned error can't find the container with id e2a1262577c5843a16ce29d02227e1eb8b51e6cfb4114edb64c9a2015add5ba8
Nov 02 01:43:31.993322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.993249    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9" event={"ID":"e6d652d1-d6ae-4e0f-b41f-f7204737b226","Type":"ContainerStarted","Data":"ee61117e1dbc885797c231651d6f26885dbdf83664ff1c1084de0433751e32a9"}
Nov 02 01:43:31.993322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.993319    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9" event={"ID":"e6d652d1-d6ae-4e0f-b41f-f7204737b226","Type":"ContainerStarted","Data":"e2a1262577c5843a16ce29d02227e1eb8b51e6cfb4114edb64c9a2015add5ba8"}
Nov 02 01:43:31.994950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.994927    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"
Nov 02 01:43:31.995081 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.994927    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8" event={"ID":"d9c36349-4646-4bcd-9b40-ede290c97edc","Type":"ContainerDied","Data":"84400a27cf5b14be292378189aff0b961ff128b67e8bfde82d1785ce35f71959"}
Nov 02 01:43:31.995081 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:31.995055    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84400a27cf5b14be292378189aff0b961ff128b67e8bfde82d1785ce35f71959"
Nov 02 01:43:32.016670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.016619    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9" podStartSLOduration=2.016604838 podStartE2EDuration="2.016604838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:32.011024021 +0000 UTC m=+1717.315956847" watchObservedRunningTime="2026-11-02 01:43:32.016604838 +0000 UTC m=+1717.321537664"
Nov 02 01:43:32.081829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.081797    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"]
Nov 02 01:43:32.103445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.103359    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"]
Nov 02 01:43:32.547434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.547396    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:43:32.614404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.614378    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:43:32.693204 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.693170    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-kube-api-access-2kr8d\" (UniqueName: \"kubernetes.io/projected/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-kube-api-access-2kr8d\") pod \"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd\" (UID: \"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd\") "
Nov 02 01:43:32.693433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.693263    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce77099f-2ea7-4032-ae62-9e089de80c22-kube-api-access-cwbn7\" (UniqueName: \"kubernetes.io/projected/ce77099f-2ea7-4032-ae62-9e089de80c22-kube-api-access-cwbn7\") pod \"ce77099f-2ea7-4032-ae62-9e089de80c22\" (UID: \"ce77099f-2ea7-4032-ae62-9e089de80c22\") "
Nov 02 01:43:32.693433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.693420    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-plugins-config-volume\") pod \"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd\" (UID: \"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd\") "
Nov 02 01:43:32.693679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.693659    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce77099f-2ea7-4032-ae62-9e089de80c22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce77099f-2ea7-4032-ae62-9e089de80c22-plugins-config-volume\") pod \"ce77099f-2ea7-4032-ae62-9e089de80c22\" (UID: \"ce77099f-2ea7-4032-ae62-9e089de80c22\") "
Nov 02 01:43:32.694236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.694207    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce77099f-2ea7-4032-ae62-9e089de80c22-plugins-config-volume" pod "ce77099f-2ea7-4032-ae62-9e089de80c22" (UID: "ce77099f-2ea7-4032-ae62-9e089de80c22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:32.694545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.694523    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce77099f-2ea7-4032-ae62-9e089de80c22-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:32.694695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.694668    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-plugins-config-volume" pod "bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd" (UID: "bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:32.695735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.695715    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-kube-api-access-2kr8d" pod "bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd" (UID: "bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd"). InnerVolumeSpecName "kube-api-access-2kr8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:32.695802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.695760    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce77099f-2ea7-4032-ae62-9e089de80c22-kube-api-access-cwbn7" pod "ce77099f-2ea7-4032-ae62-9e089de80c22" (UID: "ce77099f-2ea7-4032-ae62-9e089de80c22"). InnerVolumeSpecName "kube-api-access-cwbn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:32.796040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.796010    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kr8d\" (UniqueName: \"kubernetes.io/projected/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-kube-api-access-2kr8d\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:32.796040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.796042    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbn7\" (UniqueName: \"kubernetes.io/projected/ce77099f-2ea7-4032-ae62-9e089de80c22-kube-api-access-cwbn7\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:32.796259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:32.796055    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:33.035372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.032593    4851 generic.go:356] "Generic (PLEG): container finished" podID="ce77099f-2ea7-4032-ae62-9e089de80c22" containerID="1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9" exitCode=0
Nov 02 01:43:33.035372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.032691    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m" event={"ID":"ce77099f-2ea7-4032-ae62-9e089de80c22","Type":"ContainerDied","Data":"1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9"}
Nov 02 01:43:33.035372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.032740    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m" event={"ID":"ce77099f-2ea7-4032-ae62-9e089de80c22","Type":"ContainerDied","Data":"55e947ed39620baa0215435a935662f3349844303a538eb418511e4c784e20ee"}
Nov 02 01:43:33.035372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.032766    4851 scope.go:122] "RemoveContainer" containerID="1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9"
Nov 02 01:43:33.035372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.032932    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"
Nov 02 01:43:33.043955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.043917    4851 generic.go:356] "Generic (PLEG): container finished" podID="e6d652d1-d6ae-4e0f-b41f-f7204737b226" containerID="ee61117e1dbc885797c231651d6f26885dbdf83664ff1c1084de0433751e32a9" exitCode=0
Nov 02 01:43:33.044819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.044792    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9" event={"ID":"e6d652d1-d6ae-4e0f-b41f-f7204737b226","Type":"ContainerDied","Data":"ee61117e1dbc885797c231651d6f26885dbdf83664ff1c1084de0433751e32a9"}
Nov 02 01:43:33.051076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.051039    4851 generic.go:356] "Generic (PLEG): container finished" podID="bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd" containerID="491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06" exitCode=0
Nov 02 01:43:33.051590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.051507    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"
Nov 02 01:43:33.051813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.051530    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p" event={"ID":"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd","Type":"ContainerDied","Data":"491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06"}
Nov 02 01:43:33.051890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.051868    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p" event={"ID":"bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd","Type":"ContainerDied","Data":"d40282d4e283598aaf61ecf9e6eb15a34cc891e62f0b8481f0c9dced30da1843"}
Nov 02 01:43:33.098217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.098150    4851 scope.go:122] "RemoveContainer" containerID="1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9"
Nov 02 01:43:33.098733 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:33.098694    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9\": container with ID starting with 1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9 not found: ID does not exist" containerID="1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9"
Nov 02 01:43:33.098875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.098741    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9"} err="failed to get container status \"1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9\": rpc error: code = NotFound desc = could not find container \"1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9\": container with ID starting with 1520dca23a37a6768f7a0bc4b1dc91480b2edb370333b74f69849f242f2a56a9 not found: ID does not exist"
Nov 02 01:43:33.098875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.098767    4851 scope.go:122] "RemoveContainer" containerID="491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06"
Nov 02 01:43:33.115126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.115085    4851 scope.go:122] "RemoveContainer" containerID="491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06"
Nov 02 01:43:33.115552 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:33.115517    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06\": container with ID starting with 491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06 not found: ID does not exist" containerID="491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06"
Nov 02 01:43:33.115674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.115558    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06"} err="failed to get container status \"491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06\": rpc error: code = NotFound desc = could not find container \"491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06\": container with ID starting with 491f20af789557ed15b78508ee5e26a382da816a8b8efbe69cddc5529e33bd06 not found: ID does not exist"
Nov 02 01:43:33.118737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.118701    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"]
Nov 02 01:43:33.123084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.123060    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0bbeda-566df7b58c-t652m"]
Nov 02 01:43:33.132038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.132009    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"]
Nov 02 01:43:33.136482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:33.136456    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0bbeda-77c768864f-8kf2p"]
Nov 02 01:43:34.270354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.270328    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:34.347024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.346934    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-xks-ca\") pod \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") "
Nov 02 01:43:34.347213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.347108    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-openshift-ca\") pod \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") "
Nov 02 01:43:34.347557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.347528    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-openshift-ca" pod "e6d652d1-d6ae-4e0f-b41f-f7204737b226" (UID: "e6d652d1-d6ae-4e0f-b41f-f7204737b226"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:34.347914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.347782    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d652d1-d6ae-4e0f-b41f-f7204737b226-kube-api-access-kcscf\" (UniqueName: \"kubernetes.io/projected/e6d652d1-d6ae-4e0f-b41f-f7204737b226-kube-api-access-kcscf\") pod \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\" (UID: \"e6d652d1-d6ae-4e0f-b41f-f7204737b226\") "
Nov 02 01:43:34.349457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.349425    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-xks-ca" pod "e6d652d1-d6ae-4e0f-b41f-f7204737b226" (UID: "e6d652d1-d6ae-4e0f-b41f-f7204737b226"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:34.349557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.349514    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:34.350103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.350080    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d652d1-d6ae-4e0f-b41f-f7204737b226-kube-api-access-kcscf" pod "e6d652d1-d6ae-4e0f-b41f-f7204737b226" (UID: "e6d652d1-d6ae-4e0f-b41f-f7204737b226"). InnerVolumeSpecName "kube-api-access-kcscf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:34.450657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.450620    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcscf\" (UniqueName: \"kubernetes.io/projected/e6d652d1-d6ae-4e0f-b41f-f7204737b226-kube-api-access-kcscf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:34.450657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.450655    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6d652d1-d6ae-4e0f-b41f-f7204737b226-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:34.851406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.851364    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd" path="/var/lib/kubelet/pods/bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd/volumes"
Nov 02 01:43:34.851892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:34.851870    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce77099f-2ea7-4032-ae62-9e089de80c22" path="/var/lib/kubelet/pods/ce77099f-2ea7-4032-ae62-9e089de80c22/volumes"
Nov 02 01:43:35.073597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.073566    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"
Nov 02 01:43:35.073766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.073597    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9" event={"ID":"e6d652d1-d6ae-4e0f-b41f-f7204737b226","Type":"ContainerDied","Data":"e2a1262577c5843a16ce29d02227e1eb8b51e6cfb4114edb64c9a2015add5ba8"}
Nov 02 01:43:35.073766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.073647    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2a1262577c5843a16ce29d02227e1eb8b51e6cfb4114edb64c9a2015add5ba8"
Nov 02 01:43:35.692318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.687850    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:43:35.804616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.804529    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-credential-socket\") pod \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") "
Nov 02 01:43:35.808018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.805360    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-credential-socket" pod "b0c1d21f-cac3-4641-a6e2-92b2e12551b1" (UID: "b0c1d21f-cac3-4641-a6e2-92b2e12551b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:35.808833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.808770    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-certs" pod "b0c1d21f-cac3-4641-a6e2-92b2e12551b1" (UID: "b0c1d21f-cac3-4641-a6e2-92b2e12551b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:35.809074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.809054    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-certs\") pod \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") "
Nov 02 01:43:35.809334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.809317    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-socket\") pod \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") "
Nov 02 01:43:35.809887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.809705    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-socket" pod "b0c1d21f-cac3-4641-a6e2-92b2e12551b1" (UID: "b0c1d21f-cac3-4641-a6e2-92b2e12551b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:35.810133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.810113    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-envoy\") pod \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") "
Nov 02 01:43:35.810967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.810950    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-token\" (UniqueName: \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-token\") pod \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") "
Nov 02 01:43:35.811713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.811657    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-kube-api-access-2jw2b\" (UniqueName: \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-kube-api-access-2jw2b\") pod \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") "
Nov 02 01:43:35.812107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.812089    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-data\") pod \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") "
Nov 02 01:43:35.812824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.812806    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-podinfo\") pod \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") "
Nov 02 01:43:35.813077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.813050    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istiod-ca-cert\") pod \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\" (UID: \"b0c1d21f-cac3-4641-a6e2-92b2e12551b1\") "
Nov 02 01:43:35.814930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.814910    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:35.815124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.815109    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:35.815248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.815235    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:35.818583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.815689    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-data" pod "b0c1d21f-cac3-4641-a6e2-92b2e12551b1" (UID: "b0c1d21f-cac3-4641-a6e2-92b2e12551b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:35.818583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.818158    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istiod-ca-cert" pod "b0c1d21f-cac3-4641-a6e2-92b2e12551b1" (UID: "b0c1d21f-cac3-4641-a6e2-92b2e12551b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:35.861517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.861462    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-kube-api-access-2jw2b" pod "b0c1d21f-cac3-4641-a6e2-92b2e12551b1" (UID: "b0c1d21f-cac3-4641-a6e2-92b2e12551b1"). InnerVolumeSpecName "kube-api-access-2jw2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:35.861690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.861628    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-token" pod "b0c1d21f-cac3-4641-a6e2-92b2e12551b1" (UID: "b0c1d21f-cac3-4641-a6e2-92b2e12551b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:35.861750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.861728    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-envoy" pod "b0c1d21f-cac3-4641-a6e2-92b2e12551b1" (UID: "b0c1d21f-cac3-4641-a6e2-92b2e12551b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:35.861898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.861870    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-podinfo" pod "b0c1d21f-cac3-4641-a6e2-92b2e12551b1" (UID: "b0c1d21f-cac3-4641-a6e2-92b2e12551b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:35.925339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.921784    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:35.925339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.921812    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:35.925339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.921823    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jw2b\" (UniqueName: \"kubernetes.io/projected/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-kube-api-access-2jw2b\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:35.925339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.921835    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:35.925339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.921848    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:35.925339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:35.921857    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c1d21f-cac3-4641-a6e2-92b2e12551b1-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:36.084194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.084104    4851 generic.go:356] "Generic (PLEG): container finished" podID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerID="36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f" exitCode=0
Nov 02 01:43:36.084194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.084161    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"
Nov 02 01:43:36.084402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.084193    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" event={"ID":"b0c1d21f-cac3-4641-a6e2-92b2e12551b1","Type":"ContainerDied","Data":"36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f"}
Nov 02 01:43:36.084402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.084258    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c" event={"ID":"b0c1d21f-cac3-4641-a6e2-92b2e12551b1","Type":"ContainerDied","Data":"e5ab375b3399c57cff819dc298e52c1f68f554998fc74579a82da0102309e44f"}
Nov 02 01:43:36.084402 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.084283    4851 scope.go:122] "RemoveContainer" containerID="36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f"
Nov 02 01:43:36.096912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.096888    4851 scope.go:122] "RemoveContainer" containerID="6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194"
Nov 02 01:43:36.110543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.110519    4851 scope.go:122] "RemoveContainer" containerID="36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f"
Nov 02 01:43:36.110882 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:36.110862    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f\": container with ID starting with 36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f not found: ID does not exist" containerID="36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f"
Nov 02 01:43:36.110985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.110889    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f"} err="failed to get container status \"36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f\": rpc error: code = NotFound desc = could not find container \"36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f\": container with ID starting with 36d38bd1179f1560c29760337b43c08ecfd84d5c7eeb56b78d206b723b9ccf4f not found: ID does not exist"
Nov 02 01:43:36.110985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.110911    4851 scope.go:122] "RemoveContainer" containerID="6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194"
Nov 02 01:43:36.111231 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:36.111202    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194\": container with ID starting with 6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194 not found: ID does not exist" containerID="6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194"
Nov 02 01:43:36.111382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.111235    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194"} err="failed to get container status \"6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194\": rpc error: code = NotFound desc = could not find container \"6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194\": container with ID starting with 6f4155724d96d934c8f8ae32665b82b4a8d3892db9dbf55fb6b457ef4e4b3194 not found: ID does not exist"
Nov 02 01:43:36.113724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.113685    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"]
Nov 02 01:43:36.118377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.118353    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-zhv5c"]
Nov 02 01:43:36.849657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:36.849622    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" path="/var/lib/kubelet/pods/b0c1d21f-cac3-4641-a6e2-92b2e12551b1/volumes"
Nov 02 01:43:37.385220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:37.385182    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"]
Nov 02 01:43:37.385566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:37.385534    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy" containerID="cri-o://98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074" gracePeriod=30
Nov 02 01:43:37.436958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:37.436924    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"]
Nov 02 01:43:37.450208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:37.450176    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"]
Nov 02 01:43:38.293895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.293853    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"]
Nov 02 01:43:38.300515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.300482    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-eeeb1a-5c67w"]
Nov 02 01:43:38.351860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.351825    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"]
Nov 02 01:43:38.352670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352636    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d9c36349-4646-4bcd-9b40-ede290c97edc" containerName="revoke-keys"
Nov 02 01:43:38.352670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352664    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c36349-4646-4bcd-9b40-ede290c97edc" containerName="revoke-keys"
Nov 02 01:43:38.352845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352679    4851 cpu_manager.go:414] "containerMap: removing container" podUID="e6d652d1-d6ae-4e0f-b41f-f7204737b226" containerName="revoke-keys"
Nov 02 01:43:38.352845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352684    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d652d1-d6ae-4e0f-b41f-f7204737b226" containerName="revoke-keys"
Nov 02 01:43:38.352845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352698    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy"
Nov 02 01:43:38.352845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352703    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy"
Nov 02 01:43:38.352845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352715    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd" containerName="payload-processing"
Nov 02 01:43:38.352845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352721    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd" containerName="payload-processing"
Nov 02 01:43:38.352845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352756    4851 cpu_manager.go:414] "containerMap: removing container" podUID="ce77099f-2ea7-4032-ae62-9e089de80c22" containerName="payload-pre-processing"
Nov 02 01:43:38.352845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352763    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce77099f-2ea7-4032-ae62-9e089de80c22" containerName="payload-pre-processing"
Nov 02 01:43:38.353145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352877    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy"
Nov 02 01:43:38.353145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352886    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce77099f-2ea7-4032-ae62-9e089de80c22" containerName="payload-pre-processing"
Nov 02 01:43:38.353145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352900    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c36349-4646-4bcd-9b40-ede290c97edc" containerName="revoke-keys"
Nov 02 01:43:38.353145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352910    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d652d1-d6ae-4e0f-b41f-f7204737b226" containerName="revoke-keys"
Nov 02 01:43:38.353145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.352921    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0cabe1-d5f2-45e7-a1af-33c31cf0e3fd" containerName="payload-processing"
Nov 02 01:43:38.358474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.358448    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.369630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.368870    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:43:38.369630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.368933    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:43:38.369630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.369058    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:43:38.370076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.370054    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:43:38.370638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.370615    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"]
Nov 02 01:43:38.508371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.483631    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.508371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.483819    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.508371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.483930    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpv24\" (UniqueName: \"kubernetes.io/projected/87eda855-5122-4d86-87c6-a0fbebaa71a4-kube-api-access-xpv24\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.585814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.585719    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.586002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.585853    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpv24\" (UniqueName: \"kubernetes.io/projected/87eda855-5122-4d86-87c6-a0fbebaa71a4-kube-api-access-xpv24\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.586093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.586074    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.586430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.586411    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.589220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.589191    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.604622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.604590    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpv24\" (UniqueName: \"kubernetes.io/projected/87eda855-5122-4d86-87c6-a0fbebaa71a4-kube-api-access-xpv24\") pod \"maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.677959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.677917    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:38.851012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.850928    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52e3ca9-d2cc-4607-b160-1e963871853f" path="/var/lib/kubelet/pods/f52e3ca9-d2cc-4607-b160-1e963871853f/volumes"
Nov 02 01:43:38.976600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:38.976568    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"]
Nov 02 01:43:38.977503 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:43:38.977467    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87eda855_5122_4d86_87c6_a0fbebaa71a4.slice/crio-ac46a33d4aa42ea859eec08fbcbc11a5d2142de55824a1dfede9c0c5fb1a750b WatchSource:0}: Error finding container ac46a33d4aa42ea859eec08fbcbc11a5d2142de55824a1dfede9c0c5fb1a750b: Status 404 returned error can't find the container with id ac46a33d4aa42ea859eec08fbcbc11a5d2142de55824a1dfede9c0c5fb1a750b
Nov 02 01:43:39.152428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:39.152343    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn" event={"ID":"87eda855-5122-4d86-87c6-a0fbebaa71a4","Type":"ContainerStarted","Data":"ac46a33d4aa42ea859eec08fbcbc11a5d2142de55824a1dfede9c0c5fb1a750b"}
Nov 02 01:43:39.705195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:39.705121    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"]
Nov 02 01:43:39.710094 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:39.710071    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b34fc0-qbhhp"]
Nov 02 01:43:40.342119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:40.341200    4851 generic.go:356] "Generic (PLEG): container finished" podID="87eda855-5122-4d86-87c6-a0fbebaa71a4" containerID="8eb06e096838557dba9a22edf20c36f35f2b5984e2634e6ad32f69810dbdacb4" exitCode=0
Nov 02 01:43:40.342119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:40.341608    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn" event={"ID":"87eda855-5122-4d86-87c6-a0fbebaa71a4","Type":"ContainerDied","Data":"8eb06e096838557dba9a22edf20c36f35f2b5984e2634e6ad32f69810dbdacb4"}
Nov 02 01:43:40.754361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:40.754176    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"]
Nov 02 01:43:40.754904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:40.754491    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy" containerID="cri-o://32961c015c7e1cd2752244598c3af50ecd803b0fc75677cc9be8fce47746093a" gracePeriod=30
Nov 02 01:43:40.856144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:40.856106    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142f9451-74dc-4d9f-9210-74dd3d893ec8" path="/var/lib/kubelet/pods/142f9451-74dc-4d9f-9210-74dd3d893ec8/volumes"
Nov 02 01:43:40.903491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:40.903454    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"]
Nov 02 01:43:40.903692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:40.903660    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy" containerID="cri-o://8c84a22d516eb203c97ff0791d2743487f5249a6679deee1c6fcf44651a8547c" gracePeriod=30
Nov 02 01:43:41.011372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:40.990626    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"]
Nov 02 01:43:41.739993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:41.739969    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:41.792133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:41.792100    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-openshift-ca\") pod \"87eda855-5122-4d86-87c6-a0fbebaa71a4\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") "
Nov 02 01:43:41.792679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:41.792258    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-xks-ca\") pod \"87eda855-5122-4d86-87c6-a0fbebaa71a4\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") "
Nov 02 01:43:41.792679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:41.792379    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87eda855-5122-4d86-87c6-a0fbebaa71a4-kube-api-access-xpv24\" (UniqueName: \"kubernetes.io/projected/87eda855-5122-4d86-87c6-a0fbebaa71a4-kube-api-access-xpv24\") pod \"87eda855-5122-4d86-87c6-a0fbebaa71a4\" (UID: \"87eda855-5122-4d86-87c6-a0fbebaa71a4\") "
Nov 02 01:43:41.792679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:41.792615    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-openshift-ca" pod "87eda855-5122-4d86-87c6-a0fbebaa71a4" (UID: "87eda855-5122-4d86-87c6-a0fbebaa71a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:41.793441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:41.793419    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:41.794662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:41.794634    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-xks-ca" pod "87eda855-5122-4d86-87c6-a0fbebaa71a4" (UID: "87eda855-5122-4d86-87c6-a0fbebaa71a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:41.795106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:41.795089    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87eda855-5122-4d86-87c6-a0fbebaa71a4-kube-api-access-xpv24" pod "87eda855-5122-4d86-87c6-a0fbebaa71a4" (UID: "87eda855-5122-4d86-87c6-a0fbebaa71a4"). InnerVolumeSpecName "kube-api-access-xpv24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:41.895038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:41.895001    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87eda855-5122-4d86-87c6-a0fbebaa71a4-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:41.895038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:41.895039    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpv24\" (UniqueName: \"kubernetes.io/projected/87eda855-5122-4d86-87c6-a0fbebaa71a4-kube-api-access-xpv24\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:42.437514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.437473    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h" podUID="5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" containerName="payload-processing" containerID="cri-o://82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c" gracePeriod=25
Nov 02 01:43:42.438071 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.437988    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"
Nov 02 01:43:42.438071 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.437994    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn" event={"ID":"87eda855-5122-4d86-87c6-a0fbebaa71a4","Type":"ContainerDied","Data":"ac46a33d4aa42ea859eec08fbcbc11a5d2142de55824a1dfede9c0c5fb1a750b"}
Nov 02 01:43:42.438071 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.438038    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac46a33d4aa42ea859eec08fbcbc11a5d2142de55824a1dfede9c0c5fb1a750b"
Nov 02 01:43:42.451222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.451164    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p" podUID="14b4b0ad-6f18-410d-b545-db6f1ecd3141" containerName="payload-pre-processing" containerID="cri-o://e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba" gracePeriod=25
Nov 02 01:43:42.683893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.683866    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:43:42.710274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.710235    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-kube-api-access-hm7rn\" (UniqueName: \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-kube-api-access-hm7rn\") pod \"24da9590-78c7-472e-aa84-a64c8fda59a2\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") "
Nov 02 01:43:42.710520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.710368    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-token\" (UniqueName: \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-token\") pod \"24da9590-78c7-472e-aa84-a64c8fda59a2\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") "
Nov 02 01:43:42.710520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.710437    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24da9590-78c7-472e-aa84-a64c8fda59a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24da9590-78c7-472e-aa84-a64c8fda59a2-istiod-ca-cert\") pod \"24da9590-78c7-472e-aa84-a64c8fda59a2\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") "
Nov 02 01:43:42.710520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.710484    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-envoy\") pod \"24da9590-78c7-472e-aa84-a64c8fda59a2\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") "
Nov 02 01:43:42.710757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.710734    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-socket\") pod \"24da9590-78c7-472e-aa84-a64c8fda59a2\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") "
Nov 02 01:43:42.710843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.710789    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-data\") pod \"24da9590-78c7-472e-aa84-a64c8fda59a2\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") "
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.710942    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-credential-socket\") pod \"24da9590-78c7-472e-aa84-a64c8fda59a2\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") "
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.711028    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-podinfo\") pod \"24da9590-78c7-472e-aa84-a64c8fda59a2\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") "
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.711056    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-certs\") pod \"24da9590-78c7-472e-aa84-a64c8fda59a2\" (UID: \"24da9590-78c7-472e-aa84-a64c8fda59a2\") "
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.711184    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-socket" pod "24da9590-78c7-472e-aa84-a64c8fda59a2" (UID: "24da9590-78c7-472e-aa84-a64c8fda59a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.711445    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-credential-socket" pod "24da9590-78c7-472e-aa84-a64c8fda59a2" (UID: "24da9590-78c7-472e-aa84-a64c8fda59a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.711623    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-data" pod "24da9590-78c7-472e-aa84-a64c8fda59a2" (UID: "24da9590-78c7-472e-aa84-a64c8fda59a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.712058    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.712081    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.712096    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.713421    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-envoy" pod "24da9590-78c7-472e-aa84-a64c8fda59a2" (UID: "24da9590-78c7-472e-aa84-a64c8fda59a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.713579    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-token" pod "24da9590-78c7-472e-aa84-a64c8fda59a2" (UID: "24da9590-78c7-472e-aa84-a64c8fda59a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:42.714766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.714530    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-certs" pod "24da9590-78c7-472e-aa84-a64c8fda59a2" (UID: "24da9590-78c7-472e-aa84-a64c8fda59a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:42.715463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.714861    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24da9590-78c7-472e-aa84-a64c8fda59a2-istiod-ca-cert" pod "24da9590-78c7-472e-aa84-a64c8fda59a2" (UID: "24da9590-78c7-472e-aa84-a64c8fda59a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:42.717095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.717057    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-kube-api-access-hm7rn" pod "24da9590-78c7-472e-aa84-a64c8fda59a2" (UID: "24da9590-78c7-472e-aa84-a64c8fda59a2"). InnerVolumeSpecName "kube-api-access-hm7rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:42.718225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.718196    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-podinfo" pod "24da9590-78c7-472e-aa84-a64c8fda59a2" (UID: "24da9590-78c7-472e-aa84-a64c8fda59a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:42.814023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.813975    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:42.814556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.814168    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24da9590-78c7-472e-aa84-a64c8fda59a2-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:42.814556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.814185    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:42.814556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.814198    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24da9590-78c7-472e-aa84-a64c8fda59a2-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:42.814556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.814210    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24da9590-78c7-472e-aa84-a64c8fda59a2-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:42.814556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:42.814464    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm7rn\" (UniqueName: \"kubernetes.io/projected/24da9590-78c7-472e-aa84-a64c8fda59a2-kube-api-access-hm7rn\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:43.447985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:43.447947    4851 generic.go:356] "Generic (PLEG): container finished" podID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerID="98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074" exitCode=0
Nov 02 01:43:43.448174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:43.447998    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"
Nov 02 01:43:43.448174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:43.448033    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" event={"ID":"24da9590-78c7-472e-aa84-a64c8fda59a2","Type":"ContainerDied","Data":"98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074"}
Nov 02 01:43:43.448174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:43.448097    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7" event={"ID":"24da9590-78c7-472e-aa84-a64c8fda59a2","Type":"ContainerDied","Data":"2d5fd464cd3b002173a10b89f4718b933af404131b22fe218a8c07dbe0a3327c"}
Nov 02 01:43:43.448174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:43.448122    4851 scope.go:122] "RemoveContainer" containerID="98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074"
Nov 02 01:43:43.460565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:43.460546    4851 scope.go:122] "RemoveContainer" containerID="98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074"
Nov 02 01:43:43.460875 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:43.460853    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074\": container with ID starting with 98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074 not found: ID does not exist" containerID="98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074"
Nov 02 01:43:43.460932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:43.460885    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074"} err="failed to get container status \"98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074\": rpc error: code = NotFound desc = could not find container \"98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074\": container with ID starting with 98cadee22ebc08f3e2bb42c9e251777840b33f57cd1a03ca5006b88b4602c074 not found: ID does not exist"
Nov 02 01:43:43.471388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:43.471356    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"]
Nov 02 01:43:43.475743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:43.475721    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-vvbl7"]
Nov 02 01:43:44.850019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:44.849982    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" path="/var/lib/kubelet/pods/24da9590-78c7-472e-aa84-a64c8fda59a2/volumes"
Nov 02 01:43:45.850997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:45.850966    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"]
Nov 02 01:43:45.870111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:45.863081    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy" containerID="cri-o://65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e" gracePeriod=30
Nov 02 01:43:46.004317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:45.996407    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4" podUID="21310443-4ec4-40ae-91da-65292ae655f9" containerName="payload-processing" containerID="cri-o://cdf183c59a2b45fa584c34c8323f8ae62418537980430bb5a35f0f75e5915263" gracePeriod=25
Nov 02 01:43:46.328910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.328871    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"]
Nov 02 01:43:46.330055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.330026    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy"
Nov 02 01:43:46.330055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.330047    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy"
Nov 02 01:43:46.330455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.330092    4851 cpu_manager.go:414] "containerMap: removing container" podUID="87eda855-5122-4d86-87c6-a0fbebaa71a4" containerName="revoke-keys"
Nov 02 01:43:46.330455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.330109    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87eda855-5122-4d86-87c6-a0fbebaa71a4" containerName="revoke-keys"
Nov 02 01:43:46.330455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.330143    4851 cpu_manager.go:414] "containerMap: removing container" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy"
Nov 02 01:43:46.330455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.330148    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy"
Nov 02 01:43:46.330455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.330350    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="87eda855-5122-4d86-87c6-a0fbebaa71a4" containerName="revoke-keys"
Nov 02 01:43:46.330455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.330364    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c1d21f-cac3-4641-a6e2-92b2e12551b1" containerName="istio-proxy"
Nov 02 01:43:46.330455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.330375    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="24da9590-78c7-472e-aa84-a64c8fda59a2" containerName="istio-proxy"
Nov 02 01:43:46.334418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.334392    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.339556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.339399    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:43:46.339556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.339526    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:43:46.344817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.344329    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"]
Nov 02 01:43:46.416615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.415194    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.416615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.415456    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4pz8\" (UniqueName: \"kubernetes.io/projected/c959c5af-7076-4638-8878-ef9f20ec5902-kube-api-access-c4pz8\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.416615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.415489    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.429557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.429511    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:43:46.429919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.429890    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:43:46.484585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.484184    4851 generic.go:356] "Generic (PLEG): container finished" podID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerID="32961c015c7e1cd2752244598c3af50ecd803b0fc75677cc9be8fce47746093a" exitCode=0
Nov 02 01:43:46.484585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.484423    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" event={"ID":"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed","Type":"ContainerDied","Data":"32961c015c7e1cd2752244598c3af50ecd803b0fc75677cc9be8fce47746093a"}
Nov 02 01:43:46.484585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.484487    4851 scope.go:122] "RemoveContainer" containerID="940927f8e46c5f52d34abf6b77bcd7d1e975c6dbf3c193eb2a358f3507425244"
Nov 02 01:43:46.498926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.498894    4851 generic.go:356] "Generic (PLEG): container finished" podID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerID="8c84a22d516eb203c97ff0791d2743487f5249a6679deee1c6fcf44651a8547c" exitCode=0
Nov 02 01:43:46.500410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.500381    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" event={"ID":"4e02f029-1d4f-42ca-bca9-b963ce25eee3","Type":"ContainerDied","Data":"8c84a22d516eb203c97ff0791d2743487f5249a6679deee1c6fcf44651a8547c"}
Nov 02 01:43:46.501065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.501040    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5" event={"ID":"4e02f029-1d4f-42ca-bca9-b963ce25eee3","Type":"ContainerDied","Data":"c39da7a3ffdec8d8775ef58cb6b0334f241dcd55a0e34b7f80a1b294e77a59f2"}
Nov 02 01:43:46.501243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.501227    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c39da7a3ffdec8d8775ef58cb6b0334f241dcd55a0e34b7f80a1b294e77a59f2"
Nov 02 01:43:46.504745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.504686    4851 generic.go:356] "Generic (PLEG): container finished" podID="21310443-4ec4-40ae-91da-65292ae655f9" containerID="cdf183c59a2b45fa584c34c8323f8ae62418537980430bb5a35f0f75e5915263" exitCode=0
Nov 02 01:43:46.505570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.505549    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4" event={"ID":"21310443-4ec4-40ae-91da-65292ae655f9","Type":"ContainerDied","Data":"cdf183c59a2b45fa584c34c8323f8ae62418537980430bb5a35f0f75e5915263"}
Nov 02 01:43:46.516999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.516960    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.517182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.517159    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4pz8\" (UniqueName: \"kubernetes.io/projected/c959c5af-7076-4638-8878-ef9f20ec5902-kube-api-access-c4pz8\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.517247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.517193    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.518228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.518197    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.521740 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.521717    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:43:46.525572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.523993    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.565937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.565648    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:43:46.575495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.574644    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:43:46.577630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.577602    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4pz8\" (UniqueName: \"kubernetes.io/projected/c959c5af-7076-4638-8878-ef9f20ec5902-kube-api-access-c4pz8\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.618637    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-certs\") pod \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.618698    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-data\") pod \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.618782    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21310443-4ec4-40ae-91da-65292ae655f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21310443-4ec4-40ae-91da-65292ae655f9-plugins-config-volume\") pod \"21310443-4ec4-40ae-91da-65292ae655f9\" (UID: \"21310443-4ec4-40ae-91da-65292ae655f9\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.618818    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-socket\") pod \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.618858    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istiod-ca-cert\") pod \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.618903    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istiod-ca-cert\") pod \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.618955    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-certs\") pod \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619094    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21310443-4ec4-40ae-91da-65292ae655f9-kube-api-access-28hpp\" (UniqueName: \"kubernetes.io/projected/21310443-4ec4-40ae-91da-65292ae655f9-kube-api-access-28hpp\") pod \"21310443-4ec4-40ae-91da-65292ae655f9\" (UID: \"21310443-4ec4-40ae-91da-65292ae655f9\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619159    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-podinfo\") pod \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619208    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-envoy\") pod \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619238    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-data\") pod \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619319    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-token\" (UniqueName: \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-token\") pod \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619376    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-socket\") pod \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619415    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-token\" (UniqueName: \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-token\") pod \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619466    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-credential-socket\") pod \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619554    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-podinfo\") pod \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619611    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-kube-api-access-9fqkx\" (UniqueName: \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-kube-api-access-9fqkx\") pod \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619673    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-credential-socket\") pod \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619710    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-envoy\") pod \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\" (UID: \"4e02f029-1d4f-42ca-bca9-b963ce25eee3\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619739    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-kube-api-access-tf2z2\" (UniqueName: \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-kube-api-access-tf2z2\") pod \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\" (UID: \"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed\") "
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.619808    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istiod-ca-cert" pod "4e02f029-1d4f-42ca-bca9-b963ce25eee3" (UID: "4e02f029-1d4f-42ca-bca9-b963ce25eee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.620011    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-data" pod "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" (UID: "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:46.620202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.620148    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-socket" pod "4e02f029-1d4f-42ca-bca9-b963ce25eee3" (UID: "4e02f029-1d4f-42ca-bca9-b963ce25eee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:46.621716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.620349    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-certs" pod "4e02f029-1d4f-42ca-bca9-b963ce25eee3" (UID: "4e02f029-1d4f-42ca-bca9-b963ce25eee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:46.621716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.620835    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.621716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.620857    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.621716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.621491    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.621716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.621568    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.621716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.621101    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-data" pod "4e02f029-1d4f-42ca-bca9-b963ce25eee3" (UID: "4e02f029-1d4f-42ca-bca9-b963ce25eee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:46.621716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.621457    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21310443-4ec4-40ae-91da-65292ae655f9-plugins-config-volume" pod "21310443-4ec4-40ae-91da-65292ae655f9" (UID: "21310443-4ec4-40ae-91da-65292ae655f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:46.651137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.650654    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-certs" pod "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" (UID: "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:46.651137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.650905    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-token" pod "4e02f029-1d4f-42ca-bca9-b963ce25eee3" (UID: "4e02f029-1d4f-42ca-bca9-b963ce25eee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:46.651137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.650795    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21310443-4ec4-40ae-91da-65292ae655f9-kube-api-access-28hpp" pod "21310443-4ec4-40ae-91da-65292ae655f9" (UID: "21310443-4ec4-40ae-91da-65292ae655f9"). InnerVolumeSpecName "kube-api-access-28hpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:46.651137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.650824    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-kube-api-access-9fqkx" pod "4e02f029-1d4f-42ca-bca9-b963ce25eee3" (UID: "4e02f029-1d4f-42ca-bca9-b963ce25eee3"). InnerVolumeSpecName "kube-api-access-9fqkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:46.651137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.651096    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-credential-socket" pod "4e02f029-1d4f-42ca-bca9-b963ce25eee3" (UID: "4e02f029-1d4f-42ca-bca9-b963ce25eee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:46.651624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.651145    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-socket" pod "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" (UID: "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:46.651624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.651166    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istiod-ca-cert" pod "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" (UID: "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:46.651624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.651257    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-kube-api-access-tf2z2" pod "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" (UID: "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed"). InnerVolumeSpecName "kube-api-access-tf2z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:46.652744 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.652691    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-credential-socket" pod "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" (UID: "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:46.653781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.653753    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-podinfo" pod "4e02f029-1d4f-42ca-bca9-b963ce25eee3" (UID: "4e02f029-1d4f-42ca-bca9-b963ce25eee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:46.654066 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.654043    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-envoy" pod "4e02f029-1d4f-42ca-bca9-b963ce25eee3" (UID: "4e02f029-1d4f-42ca-bca9-b963ce25eee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:46.654394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.654374    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-envoy" pod "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" (UID: "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:46.654716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.654689    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-token" pod "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" (UID: "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:46.654819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.654791    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-podinfo" pod "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" (UID: "3f4ce5ae-b0b2-4ba7-8c04-672afae643ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:46.663110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.663078    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:46.722950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.722915    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.722950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.722947    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fqkx\" (UniqueName: \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-kube-api-access-9fqkx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.722950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.722959    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.722968    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.722977    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf2z2\" (UniqueName: \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-kube-api-access-tf2z2\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.722987    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.723000    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21310443-4ec4-40ae-91da-65292ae655f9-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.723013    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.723026    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.723039    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28hpp\" (UniqueName: \"kubernetes.io/projected/21310443-4ec4-40ae-91da-65292ae655f9-kube-api-access-28hpp\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.723052    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.723066    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.723079    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.723094    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e02f029-1d4f-42ca-bca9-b963ce25eee3-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.723103    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.723234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.723112    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:46.961260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:46.961230    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"]
Nov 02 01:43:47.105132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.105089    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" containerID="cri-o://9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443" gracePeriod=45
Nov 02 01:43:47.134805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.134764    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main" containerID="cri-o://f1f21637f77bc57322efb9d5b774991b26b35ac9202db396b7042ac74c715e21" gracePeriod=45
Nov 02 01:43:47.138651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.138593    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": dial tcp 10.132.0.90:8000: connect: connection refused"
Nov 02 01:43:47.479595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.479567    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:43:47.526523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.526487    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z" event={"ID":"c959c5af-7076-4638-8878-ef9f20ec5902","Type":"ContainerStarted","Data":"aa81fedd6741cf555cff47627fd16b7b88238a8bac77b251917fc405252c8310"}
Nov 02 01:43:47.526698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.526539    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z" event={"ID":"c959c5af-7076-4638-8878-ef9f20ec5902","Type":"ContainerStarted","Data":"dd097a21766f6c460b08e84a5e82c2800b59eb005f6e64ffec549261c20b789a"}
Nov 02 01:43:47.528171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.528145    4851 generic.go:356] "Generic (PLEG): container finished" podID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerID="f1f21637f77bc57322efb9d5b774991b26b35ac9202db396b7042ac74c715e21" exitCode=0
Nov 02 01:43:47.528307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.528205    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" event={"ID":"9fa8ee2f-b7c2-4394-8da0-b63261220c9b","Type":"ContainerDied","Data":"f1f21637f77bc57322efb9d5b774991b26b35ac9202db396b7042ac74c715e21"}
Nov 02 01:43:47.528307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.528234    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl" event={"ID":"9fa8ee2f-b7c2-4394-8da0-b63261220c9b","Type":"ContainerDied","Data":"82c2385504e75a5c309f3957a0d80ad196c65eb84bbdc4b878d2f64b69f24b89"}
Nov 02 01:43:47.528307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.528246    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c2385504e75a5c309f3957a0d80ad196c65eb84bbdc4b878d2f64b69f24b89"
Nov 02 01:43:47.528930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.528611    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:43:47.529792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.529774    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"
Nov 02 01:43:47.529882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.529815    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4" event={"ID":"21310443-4ec4-40ae-91da-65292ae655f9","Type":"ContainerDied","Data":"42e130fca4aa7af6dd9085eb24d9355f16d234aee78e4660fa55be6f2ceb9d49"}
Nov 02 01:43:47.529882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.529869    4851 scope.go:122] "RemoveContainer" containerID="cdf183c59a2b45fa584c34c8323f8ae62418537980430bb5a35f0f75e5915263"
Nov 02 01:43:47.533023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.532453    4851 generic.go:356] "Generic (PLEG): container finished" podID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerID="9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443" exitCode=0
Nov 02 01:43:47.533023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.532845    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"
Nov 02 01:43:47.533211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.533044    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" event={"ID":"bf185485-9f8b-4745-8c17-bd099588f5d4","Type":"ContainerDied","Data":"9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443"}
Nov 02 01:43:47.533211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.533070    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp" event={"ID":"bf185485-9f8b-4745-8c17-bd099588f5d4","Type":"ContainerDied","Data":"838c93390cdf044e8403aa87a057d7a78a1d9687ac1caf82794f67e8431d28dd"}
Nov 02 01:43:47.535038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.535019    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"
Nov 02 01:43:47.536134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.535703    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"
Nov 02 01:43:47.536134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.536089    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs" event={"ID":"3f4ce5ae-b0b2-4ba7-8c04-672afae643ed","Type":"ContainerDied","Data":"f4491f6fb7c5ea164fcf763e0a52816df80ff659de4a7a818cbb370f31262ba5"}
Nov 02 01:43:47.537401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.537382    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-tmp-dir\") pod \"bf185485-9f8b-4745-8c17-bd099588f5d4\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") "
Nov 02 01:43:47.537592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.537572    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-model-cache\") pod \"bf185485-9f8b-4745-8c17-bd099588f5d4\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") "
Nov 02 01:43:47.537687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.537656    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-tmp-dir" pod "bf185485-9f8b-4745-8c17-bd099588f5d4" (UID: "bf185485-9f8b-4745-8c17-bd099588f5d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:47.537745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.537696    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-dshm\") pod \"bf185485-9f8b-4745-8c17-bd099588f5d4\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") "
Nov 02 01:43:47.537745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.537732    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf185485-9f8b-4745-8c17-bd099588f5d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf185485-9f8b-4745-8c17-bd099588f5d4-tls-certs\") pod \"bf185485-9f8b-4745-8c17-bd099588f5d4\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") "
Nov 02 01:43:47.537951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.537929    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-model-cache" pod "bf185485-9f8b-4745-8c17-bd099588f5d4" (UID: "bf185485-9f8b-4745-8c17-bd099588f5d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:47.538055 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.538032    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf185485-9f8b-4745-8c17-bd099588f5d4-kube-api-access-4lr72\" (UniqueName: \"kubernetes.io/projected/bf185485-9f8b-4745-8c17-bd099588f5d4-kube-api-access-4lr72\") pod \"bf185485-9f8b-4745-8c17-bd099588f5d4\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") "
Nov 02 01:43:47.538121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.538095    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-home\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-home\") pod \"bf185485-9f8b-4745-8c17-bd099588f5d4\" (UID: \"bf185485-9f8b-4745-8c17-bd099588f5d4\") "
Nov 02 01:43:47.539185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.538951    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.539185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.538977    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.539719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.539577    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-home" pod "bf185485-9f8b-4745-8c17-bd099588f5d4" (UID: "bf185485-9f8b-4745-8c17-bd099588f5d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:47.541281 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.541243    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf185485-9f8b-4745-8c17-bd099588f5d4-tls-certs" pod "bf185485-9f8b-4745-8c17-bd099588f5d4" (UID: "bf185485-9f8b-4745-8c17-bd099588f5d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:47.541972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.541952    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-dshm" pod "bf185485-9f8b-4745-8c17-bd099588f5d4" (UID: "bf185485-9f8b-4745-8c17-bd099588f5d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:47.542197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.542175    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf185485-9f8b-4745-8c17-bd099588f5d4-kube-api-access-4lr72" pod "bf185485-9f8b-4745-8c17-bd099588f5d4" (UID: "bf185485-9f8b-4745-8c17-bd099588f5d4"). InnerVolumeSpecName "kube-api-access-4lr72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:47.550107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.549977    4851 scope.go:122] "RemoveContainer" containerID="9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443"
Nov 02 01:43:47.550226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.550102    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z" podStartSLOduration=1.550086647 podStartE2EDuration="1.550086647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:47.545535772 +0000 UTC m=+1732.850468618" watchObservedRunningTime="2026-11-02 01:43:47.550086647 +0000 UTC m=+1732.855019470"
Nov 02 01:43:47.568928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.568866    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"]
Nov 02 01:43:47.578037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.573007    4851 scope.go:122] "RemoveContainer" containerID="9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443"
Nov 02 01:43:47.578037 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:47.574829    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443\": container with ID starting with 9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443 not found: ID does not exist" containerID="9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443"
Nov 02 01:43:47.578037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.575104    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443"} err="failed to get container status \"9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443\": rpc error: code = NotFound desc = could not find container \"9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443\": container with ID starting with 9813e955b26f6308bad729e5056ed2e5bc6eb6dc102be80eeaf1d6fbd2801443 not found: ID does not exist"
Nov 02 01:43:47.578037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.575131    4851 scope.go:122] "RemoveContainer" containerID="32961c015c7e1cd2752244598c3af50ecd803b0fc75677cc9be8fce47746093a"
Nov 02 01:43:47.588644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.587359    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-k77b5"]
Nov 02 01:43:47.602966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.602916    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"]
Nov 02 01:43:47.609317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.609271    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-gvqcs"]
Nov 02 01:43:47.636564 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.636458    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p" podUID="14b4b0ad-6f18-410d-b545-db6f1ecd3141" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:43:47.636918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.636859    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h" podUID="5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:43:47.640283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.640246    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tmp-dir\") pod \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") "
Nov 02 01:43:47.640423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.640336    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-model-cache\") pod \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") "
Nov 02 01:43:47.640481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.640472    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-dshm\") pod \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") "
Nov 02 01:43:47.642106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.640572    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-kube-api-access-qhz8p\" (UniqueName: \"kubernetes.io/projected/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-kube-api-access-qhz8p\") pod \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") "
Nov 02 01:43:47.642106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.640629    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tls-certs\") pod \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") "
Nov 02 01:43:47.642106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.640682    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-home\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-home\") pod \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\" (UID: \"9fa8ee2f-b7c2-4394-8da0-b63261220c9b\") "
Nov 02 01:43:47.642106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.640807    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tmp-dir" pod "9fa8ee2f-b7c2-4394-8da0-b63261220c9b" (UID: "9fa8ee2f-b7c2-4394-8da0-b63261220c9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:47.642106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.641501    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lr72\" (UniqueName: \"kubernetes.io/projected/bf185485-9f8b-4745-8c17-bd099588f5d4-kube-api-access-4lr72\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.642106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.641523    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.642106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.641536    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.642106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.641548    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf185485-9f8b-4745-8c17-bd099588f5d4-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.642106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.641562    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf185485-9f8b-4745-8c17-bd099588f5d4-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.644073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.643544    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-model-cache" pod "9fa8ee2f-b7c2-4394-8da0-b63261220c9b" (UID: "9fa8ee2f-b7c2-4394-8da0-b63261220c9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:47.645398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.644273    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-home" pod "9fa8ee2f-b7c2-4394-8da0-b63261220c9b" (UID: "9fa8ee2f-b7c2-4394-8da0-b63261220c9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:47.646066 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.646042    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-dshm" pod "9fa8ee2f-b7c2-4394-8da0-b63261220c9b" (UID: "9fa8ee2f-b7c2-4394-8da0-b63261220c9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:47.646542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.646518    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-kube-api-access-qhz8p" pod "9fa8ee2f-b7c2-4394-8da0-b63261220c9b" (UID: "9fa8ee2f-b7c2-4394-8da0-b63261220c9b"). InnerVolumeSpecName "kube-api-access-qhz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:47.648277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.648249    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tls-certs" pod "9fa8ee2f-b7c2-4394-8da0-b63261220c9b" (UID: "9fa8ee2f-b7c2-4394-8da0-b63261220c9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:47.664635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.664571    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"]
Nov 02 01:43:47.675108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.674147    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"]
Nov 02 01:43:47.675108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.674457    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" podUID="5126a114-46de-4898-8088-c19a89fb51f1" containerName="istio-proxy" containerID="cri-o://69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229" gracePeriod=30
Nov 02 01:43:47.682105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.682081    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2589b2-b8468bcc9-85zc4"]
Nov 02 01:43:47.689824 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:47.689794    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:43:47.743411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.743335    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.743411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.743363    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhz8p\" (UniqueName: \"kubernetes.io/projected/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-kube-api-access-qhz8p\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.743411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.743374    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.743411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.743384    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:47.743411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:47.743392    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fa8ee2f-b7c2-4394-8da0-b63261220c9b-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:48.063636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.063558    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"]
Nov 02 01:43:48.071417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.071375    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-2-w2-47dd19-kserve-5cff7c7f99-jfdkp"]
Nov 02 01:43:48.195840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.195805    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"]
Nov 02 01:43:48.196557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196543    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main"
Nov 02 01:43:48.196626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196559    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main"
Nov 02 01:43:48.196626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196577    4851 cpu_manager.go:414] "containerMap: removing container" podUID="21310443-4ec4-40ae-91da-65292ae655f9" containerName="payload-processing"
Nov 02 01:43:48.196626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196582    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21310443-4ec4-40ae-91da-65292ae655f9" containerName="payload-processing"
Nov 02 01:43:48.196626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196601    4851 cpu_manager.go:414] "containerMap: removing container" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy"
Nov 02 01:43:48.196626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196606    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy"
Nov 02 01:43:48.196790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196641    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main"
Nov 02 01:43:48.196790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196647    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main"
Nov 02 01:43:48.196790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196658    4851 cpu_manager.go:414] "containerMap: removing container" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy"
Nov 02 01:43:48.196790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196664    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy"
Nov 02 01:43:48.196790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196670    4851 cpu_manager.go:414] "containerMap: removing container" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy"
Nov 02 01:43:48.196790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196674    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy"
Nov 02 01:43:48.196790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196770    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" containerName="main"
Nov 02 01:43:48.196790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196779    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy"
Nov 02 01:43:48.196790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196784    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" containerName="istio-proxy"
Nov 02 01:43:48.196790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196791    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" containerName="main"
Nov 02 01:43:48.197108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196799    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="21310443-4ec4-40ae-91da-65292ae655f9" containerName="payload-processing"
Nov 02 01:43:48.197108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.196806    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" containerName="istio-proxy"
Nov 02 01:43:48.200865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.200837    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.228576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.228477    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"]
Nov 02 01:43:48.376636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.376531    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.376860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.376835    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt82x\" (UniqueName: \"kubernetes.io/projected/d267cca5-7ae8-452a-bf78-dc59b87bf208-kube-api-access-lt82x\") pod \"maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.376920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.376904    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.478224 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.478188    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt82x\" (UniqueName: \"kubernetes.io/projected/d267cca5-7ae8-452a-bf78-dc59b87bf208-kube-api-access-lt82x\") pod \"maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.478466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.478271    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.478466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.478374    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.479518 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.479489    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.481460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.481431    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.483189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.483155    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"]
Nov 02 01:43:48.501383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.494715    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt82x\" (UniqueName: \"kubernetes.io/projected/d267cca5-7ae8-452a-bf78-dc59b87bf208-kube-api-access-lt82x\") pod \"maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.531978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.531948    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:48.565181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.554143    4851 generic.go:356] "Generic (PLEG): container finished" podID="c959c5af-7076-4638-8878-ef9f20ec5902" containerID="aa81fedd6741cf555cff47627fd16b7b88238a8bac77b251917fc405252c8310" exitCode=0
Nov 02 01:43:48.565181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.554345    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z" event={"ID":"c959c5af-7076-4638-8878-ef9f20ec5902","Type":"ContainerDied","Data":"aa81fedd6741cf555cff47627fd16b7b88238a8bac77b251917fc405252c8310"}
Nov 02 01:43:48.571946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.571918    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"
Nov 02 01:43:48.617645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.617606    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"]
Nov 02 01:43:48.620994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.620963    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/simulator-w2-47dd19-kserve-744b66f76-sghdl"]
Nov 02 01:43:48.946384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.940952    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21310443-4ec4-40ae-91da-65292ae655f9" path="/var/lib/kubelet/pods/21310443-4ec4-40ae-91da-65292ae655f9/volumes"
Nov 02 01:43:48.966552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.966521    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4ce5ae-b0b2-4ba7-8c04-672afae643ed" path="/var/lib/kubelet/pods/3f4ce5ae-b0b2-4ba7-8c04-672afae643ed/volumes"
Nov 02 01:43:48.995317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.981906    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e02f029-1d4f-42ca-bca9-b963ce25eee3" path="/var/lib/kubelet/pods/4e02f029-1d4f-42ca-bca9-b963ce25eee3/volumes"
Nov 02 01:43:49.007318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:48.999149    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa8ee2f-b7c2-4394-8da0-b63261220c9b" path="/var/lib/kubelet/pods/9fa8ee2f-b7c2-4394-8da0-b63261220c9b/volumes"
Nov 02 01:43:49.056315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.018278    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf185485-9f8b-4745-8c17-bd099588f5d4" path="/var/lib/kubelet/pods/bf185485-9f8b-4745-8c17-bd099588f5d4/volumes"
Nov 02 01:43:49.084647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.081984    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"]
Nov 02 01:43:49.581708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.581672    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz" event={"ID":"d267cca5-7ae8-452a-bf78-dc59b87bf208","Type":"ContainerStarted","Data":"f69ba84197c25f21e3a608013a9a93b56894f9f5355d38b781388c776edb7ca9"}
Nov 02 01:43:49.581890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.581725    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz" event={"ID":"d267cca5-7ae8-452a-bf78-dc59b87bf208","Type":"ContainerStarted","Data":"ff89d55063278c98999388d5b3892a77942fcb415e857a4ff82188fbb618485e"}
Nov 02 01:43:49.604644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.604538    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz" podStartSLOduration=1.6045237700000001 podStartE2EDuration="1.60452377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:49.602676062 +0000 UTC m=+1734.907608889" watchObservedRunningTime="2026-11-02 01:43:49.60452377 +0000 UTC m=+1734.909456596"
Nov 02 01:43:49.754056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.754030    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:49.820512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.820477    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c959c5af-7076-4638-8878-ef9f20ec5902-kube-api-access-c4pz8\" (UniqueName: \"kubernetes.io/projected/c959c5af-7076-4638-8878-ef9f20ec5902-kube-api-access-c4pz8\") pod \"c959c5af-7076-4638-8878-ef9f20ec5902\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") "
Nov 02 01:43:49.820690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.820608    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-xks-ca\") pod \"c959c5af-7076-4638-8878-ef9f20ec5902\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") "
Nov 02 01:43:49.820780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.820687    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-openshift-ca\") pod \"c959c5af-7076-4638-8878-ef9f20ec5902\" (UID: \"c959c5af-7076-4638-8878-ef9f20ec5902\") "
Nov 02 01:43:49.821118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.821089    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-openshift-ca" pod "c959c5af-7076-4638-8878-ef9f20ec5902" (UID: "c959c5af-7076-4638-8878-ef9f20ec5902"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:49.821535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.821513    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:49.822951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.822930    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c959c5af-7076-4638-8878-ef9f20ec5902-kube-api-access-c4pz8" pod "c959c5af-7076-4638-8878-ef9f20ec5902" (UID: "c959c5af-7076-4638-8878-ef9f20ec5902"). InnerVolumeSpecName "kube-api-access-c4pz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:49.823350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.823334    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-xks-ca" pod "c959c5af-7076-4638-8878-ef9f20ec5902" (UID: "c959c5af-7076-4638-8878-ef9f20ec5902"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:49.923882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.923681    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4pz8\" (UniqueName: \"kubernetes.io/projected/c959c5af-7076-4638-8878-ef9f20ec5902-kube-api-access-c4pz8\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:49.923882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:49.923714    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c959c5af-7076-4638-8878-ef9f20ec5902-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:50.588909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:50.588846    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"]
Nov 02 01:43:50.598778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:50.598743    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2589b2-sjjs9"]
Nov 02 01:43:50.600499 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:50.600475    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"
Nov 02 01:43:50.600637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:50.600502    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z" event={"ID":"c959c5af-7076-4638-8878-ef9f20ec5902","Type":"ContainerDied","Data":"dd097a21766f6c460b08e84a5e82c2800b59eb005f6e64ffec549261c20b789a"}
Nov 02 01:43:50.600637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:50.600593    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd097a21766f6c460b08e84a5e82c2800b59eb005f6e64ffec549261c20b789a"
Nov 02 01:43:50.602422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:50.602393    4851 generic.go:356] "Generic (PLEG): container finished" podID="d267cca5-7ae8-452a-bf78-dc59b87bf208" containerID="f69ba84197c25f21e3a608013a9a93b56894f9f5355d38b781388c776edb7ca9" exitCode=0
Nov 02 01:43:50.602559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:50.602473    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz" event={"ID":"d267cca5-7ae8-452a-bf78-dc59b87bf208","Type":"ContainerDied","Data":"f69ba84197c25f21e3a608013a9a93b56894f9f5355d38b781388c776edb7ca9"}
Nov 02 01:43:50.859569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:50.859486    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d652d1-d6ae-4e0f-b41f-f7204737b226" path="/var/lib/kubelet/pods/e6d652d1-d6ae-4e0f-b41f-f7204737b226/volumes"
Nov 02 01:43:51.192495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.192461    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:43:51.262615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.262572    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-istio-token\" (UniqueName: \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-istio-token\") pod \"20b253b7-6535-426d-814c-9c509d7c2992\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") "
Nov 02 01:43:51.262822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.262622    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-envoy\") pod \"20b253b7-6535-426d-814c-9c509d7c2992\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") "
Nov 02 01:43:51.262822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.262721    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b253b7-6535-426d-814c-9c509d7c2992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b253b7-6535-426d-814c-9c509d7c2992-istiod-ca-cert\") pod \"20b253b7-6535-426d-814c-9c509d7c2992\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") "
Nov 02 01:43:51.262822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.262762    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-data\") pod \"20b253b7-6535-426d-814c-9c509d7c2992\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") "
Nov 02 01:43:51.263006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.262880    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-kube-api-access-qxwc6\" (UniqueName: \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-kube-api-access-qxwc6\") pod \"20b253b7-6535-426d-814c-9c509d7c2992\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") "
Nov 02 01:43:51.263006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.262988    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-credential-socket\") pod \"20b253b7-6535-426d-814c-9c509d7c2992\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") "
Nov 02 01:43:51.263208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.263184    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-socket\") pod \"20b253b7-6535-426d-814c-9c509d7c2992\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") "
Nov 02 01:43:51.263279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.263228    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-certs\") pod \"20b253b7-6535-426d-814c-9c509d7c2992\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") "
Nov 02 01:43:51.263360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.263329    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20b253b7-6535-426d-814c-9c509d7c2992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b253b7-6535-426d-814c-9c509d7c2992-istio-podinfo\") pod \"20b253b7-6535-426d-814c-9c509d7c2992\" (UID: \"20b253b7-6535-426d-814c-9c509d7c2992\") "
Nov 02 01:43:51.263484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.263399    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-credential-socket" pod "20b253b7-6535-426d-814c-9c509d7c2992" (UID: "20b253b7-6535-426d-814c-9c509d7c2992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:51.263690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.263600    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-data" pod "20b253b7-6535-426d-814c-9c509d7c2992" (UID: "20b253b7-6535-426d-814c-9c509d7c2992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:51.264152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.264132    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:51.264259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.264156    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:51.264435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.264416    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-socket" pod "20b253b7-6535-426d-814c-9c509d7c2992" (UID: "20b253b7-6535-426d-814c-9c509d7c2992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:51.264513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.264484    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-certs" pod "20b253b7-6535-426d-814c-9c509d7c2992" (UID: "20b253b7-6535-426d-814c-9c509d7c2992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:51.264648 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.264630    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b253b7-6535-426d-814c-9c509d7c2992-istiod-ca-cert" pod "20b253b7-6535-426d-814c-9c509d7c2992" (UID: "20b253b7-6535-426d-814c-9c509d7c2992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:51.265581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.265554    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-istio-token" pod "20b253b7-6535-426d-814c-9c509d7c2992" (UID: "20b253b7-6535-426d-814c-9c509d7c2992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:51.266825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.266800    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-envoy" pod "20b253b7-6535-426d-814c-9c509d7c2992" (UID: "20b253b7-6535-426d-814c-9c509d7c2992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:51.266922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.266812    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20b253b7-6535-426d-814c-9c509d7c2992-istio-podinfo" pod "20b253b7-6535-426d-814c-9c509d7c2992" (UID: "20b253b7-6535-426d-814c-9c509d7c2992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:51.266922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.266836    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-kube-api-access-qxwc6" pod "20b253b7-6535-426d-814c-9c509d7c2992" (UID: "20b253b7-6535-426d-814c-9c509d7c2992"). InnerVolumeSpecName "kube-api-access-qxwc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:51.365987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.365953    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:51.365987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.365984    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:51.365987 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.365995    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b253b7-6535-426d-814c-9c509d7c2992-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:51.366229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.366005    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:51.366229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.366014    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b253b7-6535-426d-814c-9c509d7c2992-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:51.366229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.366022    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b253b7-6535-426d-814c-9c509d7c2992-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:51.366229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.366031    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxwc6\" (UniqueName: \"kubernetes.io/projected/20b253b7-6535-426d-814c-9c509d7c2992-kube-api-access-qxwc6\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:51.684048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.683992    4851 generic.go:356] "Generic (PLEG): container finished" podID="20b253b7-6535-426d-814c-9c509d7c2992" containerID="65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e" exitCode=0
Nov 02 01:43:51.684556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.684506    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"
Nov 02 01:43:51.699654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.699611    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" event={"ID":"20b253b7-6535-426d-814c-9c509d7c2992","Type":"ContainerDied","Data":"65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e"}
Nov 02 01:43:51.699866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.699684    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn" event={"ID":"20b253b7-6535-426d-814c-9c509d7c2992","Type":"ContainerDied","Data":"49f4fe0ad5365351ba026094acdade8bf37ed8de20cc8cc8a1d36dc5c6e310a9"}
Nov 02 01:43:51.699866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.699707    4851 scope.go:122] "RemoveContainer" containerID="65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e"
Nov 02 01:43:51.720485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.720453    4851 scope.go:122] "RemoveContainer" containerID="40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36"
Nov 02 01:43:51.730656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.730629    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"]
Nov 02 01:43:51.735910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.735878    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-eeeb1a-openshift-default-988dcd5bf-qdwwn"]
Nov 02 01:43:51.741137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.741118    4851 scope.go:122] "RemoveContainer" containerID="65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e"
Nov 02 01:43:51.741541 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:51.741521    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e\": container with ID starting with 65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e not found: ID does not exist" containerID="65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e"
Nov 02 01:43:51.741631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.741554    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e"} err="failed to get container status \"65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e\": rpc error: code = NotFound desc = could not find container \"65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e\": container with ID starting with 65ec13bf8e0f0365d9dc147c7f0c65e209bfdf5fb2b729519f70d77c1352911e not found: ID does not exist"
Nov 02 01:43:51.741631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.741580    4851 scope.go:122] "RemoveContainer" containerID="40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36"
Nov 02 01:43:51.741870 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:51.741851    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36\": container with ID starting with 40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36 not found: ID does not exist" containerID="40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36"
Nov 02 01:43:51.741911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.741876    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36"} err="failed to get container status \"40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36\": rpc error: code = NotFound desc = could not find container \"40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36\": container with ID starting with 40397a9277426de0b6224465c1b02e79a37b7d5bd1087fd7387c724a9e332c36 not found: ID does not exist"
Nov 02 01:43:51.895729 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.895702    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:51.980263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.980228    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-openshift-ca\") pod \"d267cca5-7ae8-452a-bf78-dc59b87bf208\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") "
Nov 02 01:43:51.980480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.980282    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d267cca5-7ae8-452a-bf78-dc59b87bf208-kube-api-access-lt82x\" (UniqueName: \"kubernetes.io/projected/d267cca5-7ae8-452a-bf78-dc59b87bf208-kube-api-access-lt82x\") pod \"d267cca5-7ae8-452a-bf78-dc59b87bf208\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") "
Nov 02 01:43:51.980638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.980615    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-xks-ca\") pod \"d267cca5-7ae8-452a-bf78-dc59b87bf208\" (UID: \"d267cca5-7ae8-452a-bf78-dc59b87bf208\") "
Nov 02 01:43:51.980716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.980634    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-openshift-ca" pod "d267cca5-7ae8-452a-bf78-dc59b87bf208" (UID: "d267cca5-7ae8-452a-bf78-dc59b87bf208"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:51.981202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.981180    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:51.983231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.983211    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d267cca5-7ae8-452a-bf78-dc59b87bf208-kube-api-access-lt82x" pod "d267cca5-7ae8-452a-bf78-dc59b87bf208" (UID: "d267cca5-7ae8-452a-bf78-dc59b87bf208"). InnerVolumeSpecName "kube-api-access-lt82x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:51.983362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:51.983345    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-xks-ca" pod "d267cca5-7ae8-452a-bf78-dc59b87bf208" (UID: "d267cca5-7ae8-452a-bf78-dc59b87bf208"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:52.082329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.082259    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt82x\" (UniqueName: \"kubernetes.io/projected/d267cca5-7ae8-452a-bf78-dc59b87bf208-kube-api-access-lt82x\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:52.082329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.082310    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d267cca5-7ae8-452a-bf78-dc59b87bf208-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:52.694784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.694747    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"
Nov 02 01:43:52.694784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.694771    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz" event={"ID":"d267cca5-7ae8-452a-bf78-dc59b87bf208","Type":"ContainerDied","Data":"ff89d55063278c98999388d5b3892a77942fcb415e857a4ff82188fbb618485e"}
Nov 02 01:43:52.695259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.694822    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff89d55063278c98999388d5b3892a77942fcb415e857a4ff82188fbb618485e"
Nov 02 01:43:52.851870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.851833    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b253b7-6535-426d-814c-9c509d7c2992" path="/var/lib/kubelet/pods/20b253b7-6535-426d-814c-9c509d7c2992/volumes"
Nov 02 01:43:52.854929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.854898    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"]
Nov 02 01:43:52.855657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855641    4851 cpu_manager.go:414] "containerMap: removing container" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy"
Nov 02 01:43:52.855731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855658    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy"
Nov 02 01:43:52.855731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855677    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d267cca5-7ae8-452a-bf78-dc59b87bf208" containerName="revoke-keys"
Nov 02 01:43:52.855731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855682    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d267cca5-7ae8-452a-bf78-dc59b87bf208" containerName="revoke-keys"
Nov 02 01:43:52.855731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855702    4851 cpu_manager.go:414] "containerMap: removing container" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy"
Nov 02 01:43:52.855731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855707    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy"
Nov 02 01:43:52.855731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855720    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c959c5af-7076-4638-8878-ef9f20ec5902" containerName="revoke-keys"
Nov 02 01:43:52.855731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855724    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c959c5af-7076-4638-8878-ef9f20ec5902" containerName="revoke-keys"
Nov 02 01:43:52.855945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855835    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c959c5af-7076-4638-8878-ef9f20ec5902" containerName="revoke-keys"
Nov 02 01:43:52.855945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855850    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy"
Nov 02 01:43:52.855945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.855857    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d267cca5-7ae8-452a-bf78-dc59b87bf208" containerName="revoke-keys"
Nov 02 01:43:52.860018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.859990    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:52.867099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.867061    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"]
Nov 02 01:43:52.868168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.868145    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b253b7-6535-426d-814c-9c509d7c2992" containerName="istio-proxy"
Nov 02 01:43:52.870442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.870403    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:43:52.870586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.870563    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:43:52.870791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.870752    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:43:52.871476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.870303    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:43:52.873582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.873483    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:52.875042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.874980    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"]
Nov 02 01:43:52.890637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.882680    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"]
Nov 02 01:43:52.893917 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.893882    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vcpp\" (UniqueName: \"kubernetes.io/projected/259f5008-f958-49b5-b794-5c525b55040a-kube-api-access-9vcpp\") pod \"maas-api-revoke-sub-ac77b7df-76sfq\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:52.894145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.894008    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40220884-76qft\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:52.894145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.894081    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259f5008-f958-49b5-b794-5c525b55040a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac77b7df-76sfq\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:52.895432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.894244    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9b6s\" (UniqueName: \"kubernetes.io/projected/512964a3-c7ba-4bea-ab47-ab1566234737-kube-api-access-b9b6s\") pod \"maas-api-revoke-sub-40220884-76qft\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:52.895432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.894314    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40220884-76qft\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:52.895432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.894391    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259f5008-f958-49b5-b794-5c525b55040a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac77b7df-76sfq\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:52.996105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.996022    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259f5008-f958-49b5-b794-5c525b55040a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac77b7df-76sfq\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:52.996263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.996142    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vcpp\" (UniqueName: \"kubernetes.io/projected/259f5008-f958-49b5-b794-5c525b55040a-kube-api-access-9vcpp\") pod \"maas-api-revoke-sub-ac77b7df-76sfq\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:52.996263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.996246    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40220884-76qft\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:52.996396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.996339    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259f5008-f958-49b5-b794-5c525b55040a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac77b7df-76sfq\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:52.996523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.996505    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9b6s\" (UniqueName: \"kubernetes.io/projected/512964a3-c7ba-4bea-ab47-ab1566234737-kube-api-access-b9b6s\") pod \"maas-api-revoke-sub-40220884-76qft\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:52.996596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.996564    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40220884-76qft\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:52.997228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.996986    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40220884-76qft\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:52.997228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.997016    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259f5008-f958-49b5-b794-5c525b55040a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac77b7df-76sfq\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:52.999595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:52.999573    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259f5008-f958-49b5-b794-5c525b55040a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac77b7df-76sfq\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:53.002862 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.002837    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40220884-76qft\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:53.008558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.008536    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vcpp\" (UniqueName: \"kubernetes.io/projected/259f5008-f958-49b5-b794-5c525b55040a-kube-api-access-9vcpp\") pod \"maas-api-revoke-sub-ac77b7df-76sfq\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:53.008654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.008558    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9b6s\" (UniqueName: \"kubernetes.io/projected/512964a3-c7ba-4bea-ab47-ab1566234737-kube-api-access-b9b6s\") pod \"maas-api-revoke-sub-40220884-76qft\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:53.045324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.045268    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:43:53.097741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.097710    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5126a114-46de-4898-8088-c19a89fb51f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5126a114-46de-4898-8088-c19a89fb51f1-istio-podinfo\") pod \"5126a114-46de-4898-8088-c19a89fb51f1\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") "
Nov 02 01:43:53.097949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.097769    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-istio-token\" (UniqueName: \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-istio-token\") pod \"5126a114-46de-4898-8088-c19a89fb51f1\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") "
Nov 02 01:43:53.097949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.097822    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-socket\") pod \"5126a114-46de-4898-8088-c19a89fb51f1\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") "
Nov 02 01:43:53.097949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.097896    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-certs\") pod \"5126a114-46de-4898-8088-c19a89fb51f1\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") "
Nov 02 01:43:53.097949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.097924    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-envoy\") pod \"5126a114-46de-4898-8088-c19a89fb51f1\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") "
Nov 02 01:43:53.098236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.098017    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-credential-socket\") pod \"5126a114-46de-4898-8088-c19a89fb51f1\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") "
Nov 02 01:43:53.098236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.098230    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-data\") pod \"5126a114-46de-4898-8088-c19a89fb51f1\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") "
Nov 02 01:43:53.098372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.098257    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5126a114-46de-4898-8088-c19a89fb51f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5126a114-46de-4898-8088-c19a89fb51f1-istiod-ca-cert\") pod \"5126a114-46de-4898-8088-c19a89fb51f1\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") "
Nov 02 01:43:53.098372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.098328    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-kube-api-access-5j58b\" (UniqueName: \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-kube-api-access-5j58b\") pod \"5126a114-46de-4898-8088-c19a89fb51f1\" (UID: \"5126a114-46de-4898-8088-c19a89fb51f1\") "
Nov 02 01:43:53.098479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.098371    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-credential-socket" pod "5126a114-46de-4898-8088-c19a89fb51f1" (UID: "5126a114-46de-4898-8088-c19a89fb51f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:53.098540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.098497    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-certs" pod "5126a114-46de-4898-8088-c19a89fb51f1" (UID: "5126a114-46de-4898-8088-c19a89fb51f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:53.098638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.098604    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-data" pod "5126a114-46de-4898-8088-c19a89fb51f1" (UID: "5126a114-46de-4898-8088-c19a89fb51f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:53.098839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.098814    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5126a114-46de-4898-8088-c19a89fb51f1-istiod-ca-cert" pod "5126a114-46de-4898-8088-c19a89fb51f1" (UID: "5126a114-46de-4898-8088-c19a89fb51f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:53.099077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.099058    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-socket" pod "5126a114-46de-4898-8088-c19a89fb51f1" (UID: "5126a114-46de-4898-8088-c19a89fb51f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:53.099262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.099239    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:53.099262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.099264    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5126a114-46de-4898-8088-c19a89fb51f1-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:53.099484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.099279    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:53.099484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.099313    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:53.099484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.099325    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:53.102105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.100576    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5126a114-46de-4898-8088-c19a89fb51f1-istio-podinfo" pod "5126a114-46de-4898-8088-c19a89fb51f1" (UID: "5126a114-46de-4898-8088-c19a89fb51f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:43:53.108554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.102380    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-kube-api-access-5j58b" pod "5126a114-46de-4898-8088-c19a89fb51f1" (UID: "5126a114-46de-4898-8088-c19a89fb51f1"). InnerVolumeSpecName "kube-api-access-5j58b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:53.108554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.102720    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-istio-token" pod "5126a114-46de-4898-8088-c19a89fb51f1" (UID: "5126a114-46de-4898-8088-c19a89fb51f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:53.108554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.103117    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-envoy" pod "5126a114-46de-4898-8088-c19a89fb51f1" (UID: "5126a114-46de-4898-8088-c19a89fb51f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:43:53.237323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.235606    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:53.237323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.236567    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:53.239057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.239024    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j58b\" (UniqueName: \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-kube-api-access-5j58b\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:53.239241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.239062    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5126a114-46de-4898-8088-c19a89fb51f1-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:53.239241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.239077    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5126a114-46de-4898-8088-c19a89fb51f1-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:53.239241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.239091    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5126a114-46de-4898-8088-c19a89fb51f1-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:53.484196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.484124    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns" podUID="4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" containerName="payload-pre-processing" containerID="cri-o://0be195cf93747fd247b588c8c82c5ba4bbccdcefe86e3128c35589133c548653" gracePeriod=25
Nov 02 01:43:53.588251 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:43:53.588215    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259f5008_f958_49b5_b794_5c525b55040a.slice/crio-a5a42a032a5b3d4ae37cc2ef19a91fc96ed91ad70d696801b9b481fc927da66a WatchSource:0}: Error finding container a5a42a032a5b3d4ae37cc2ef19a91fc96ed91ad70d696801b9b481fc927da66a: Status 404 returned error can't find the container with id a5a42a032a5b3d4ae37cc2ef19a91fc96ed91ad70d696801b9b481fc927da66a
Nov 02 01:43:53.589121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.588964    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"]
Nov 02 01:43:53.594019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.593983    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"]
Nov 02 01:43:53.596614 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:43:53.596591    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512964a3_c7ba_4bea_ab47_ab1566234737.slice/crio-0bcc820f05f178999cc24e029f4e51f6cc80eb44ddb2c973c1892bb0532c5eb8 WatchSource:0}: Error finding container 0bcc820f05f178999cc24e029f4e51f6cc80eb44ddb2c973c1892bb0532c5eb8: Status 404 returned error can't find the container with id 0bcc820f05f178999cc24e029f4e51f6cc80eb44ddb2c973c1892bb0532c5eb8
Nov 02 01:43:53.707339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.707302    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft" event={"ID":"512964a3-c7ba-4bea-ab47-ab1566234737","Type":"ContainerStarted","Data":"0bcc820f05f178999cc24e029f4e51f6cc80eb44ddb2c973c1892bb0532c5eb8"}
Nov 02 01:43:53.708902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.708876    4851 generic.go:356] "Generic (PLEG): container finished" podID="5126a114-46de-4898-8088-c19a89fb51f1" containerID="69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229" exitCode=0
Nov 02 01:43:53.709005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.708924    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"
Nov 02 01:43:53.709005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.708959    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" event={"ID":"5126a114-46de-4898-8088-c19a89fb51f1","Type":"ContainerDied","Data":"69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229"}
Nov 02 01:43:53.709114 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.709007    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q" event={"ID":"5126a114-46de-4898-8088-c19a89fb51f1","Type":"ContainerDied","Data":"725bf46f97aca4038f33f42e2c1f79896f4d8ac07e5b70e2b4f8c1dd0a447db4"}
Nov 02 01:43:53.709114 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.709031    4851 scope.go:122] "RemoveContainer" containerID="69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229"
Nov 02 01:43:53.710449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.710417    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq" event={"ID":"259f5008-f958-49b5-b794-5c525b55040a","Type":"ContainerStarted","Data":"a5a42a032a5b3d4ae37cc2ef19a91fc96ed91ad70d696801b9b481fc927da66a"}
Nov 02 01:43:53.722522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.722497    4851 scope.go:122] "RemoveContainer" containerID="69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229"
Nov 02 01:43:53.722825 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:43:53.722808    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229\": container with ID starting with 69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229 not found: ID does not exist" containerID="69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229"
Nov 02 01:43:53.722874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.722832    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229"} err="failed to get container status \"69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229\": rpc error: code = NotFound desc = could not find container \"69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229\": container with ID starting with 69f9e0b24264df7a430e8b776653f3a288af7d5bb15a3510c5b822000da07229 not found: ID does not exist"
Nov 02 01:43:53.736117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.736078    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"]
Nov 02 01:43:53.741104 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.741078    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-fvn9q"]
Nov 02 01:43:53.990607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:53.990272    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns" podUID="4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:43:54.720464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:54.720426    4851 generic.go:356] "Generic (PLEG): container finished" podID="259f5008-f958-49b5-b794-5c525b55040a" containerID="e93ec1f5bd8fd2930209b7965ffcc4f93d7ff431f89d4a2a1e52d4ce7e51274a" exitCode=0
Nov 02 01:43:54.720924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:54.720507    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq" event={"ID":"259f5008-f958-49b5-b794-5c525b55040a","Type":"ContainerDied","Data":"e93ec1f5bd8fd2930209b7965ffcc4f93d7ff431f89d4a2a1e52d4ce7e51274a"}
Nov 02 01:43:54.722226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:54.722204    4851 generic.go:356] "Generic (PLEG): container finished" podID="512964a3-c7ba-4bea-ab47-ab1566234737" containerID="ff547042f13b571884c94b2901c7961ce487b7ddfea54f5bda51554fc2a61cf4" exitCode=0
Nov 02 01:43:54.722353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:54.722283    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft" event={"ID":"512964a3-c7ba-4bea-ab47-ab1566234737","Type":"ContainerDied","Data":"ff547042f13b571884c94b2901c7961ce487b7ddfea54f5bda51554fc2a61cf4"}
Nov 02 01:43:54.851059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:54.850986    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5126a114-46de-4898-8088-c19a89fb51f1" path="/var/lib/kubelet/pods/5126a114-46de-4898-8088-c19a89fb51f1/volumes"
Nov 02 01:43:55.935643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:55.935615    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:55.988049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:55.987979    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:56.101090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.101056    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512964a3-c7ba-4bea-ab47-ab1566234737-kube-api-access-b9b6s\" (UniqueName: \"kubernetes.io/projected/512964a3-c7ba-4bea-ab47-ab1566234737-kube-api-access-b9b6s\") pod \"512964a3-c7ba-4bea-ab47-ab1566234737\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") "
Nov 02 01:43:56.101248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.101109    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-xks-ca\") pod \"512964a3-c7ba-4bea-ab47-ab1566234737\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") "
Nov 02 01:43:56.101248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.101241    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-openshift-ca\") pod \"512964a3-c7ba-4bea-ab47-ab1566234737\" (UID: \"512964a3-c7ba-4bea-ab47-ab1566234737\") "
Nov 02 01:43:56.101367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.101325    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259f5008-f958-49b5-b794-5c525b55040a-kube-api-access-9vcpp\" (UniqueName: \"kubernetes.io/projected/259f5008-f958-49b5-b794-5c525b55040a-kube-api-access-9vcpp\") pod \"259f5008-f958-49b5-b794-5c525b55040a\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") "
Nov 02 01:43:56.101419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.101402    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259f5008-f958-49b5-b794-5c525b55040a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259f5008-f958-49b5-b794-5c525b55040a-maas-api-openshift-ca\") pod \"259f5008-f958-49b5-b794-5c525b55040a\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") "
Nov 02 01:43:56.101507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.101488    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/259f5008-f958-49b5-b794-5c525b55040a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259f5008-f958-49b5-b794-5c525b55040a-maas-api-xks-ca\") pod \"259f5008-f958-49b5-b794-5c525b55040a\" (UID: \"259f5008-f958-49b5-b794-5c525b55040a\") "
Nov 02 01:43:56.101731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.101673    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-openshift-ca" pod "512964a3-c7ba-4bea-ab47-ab1566234737" (UID: "512964a3-c7ba-4bea-ab47-ab1566234737"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:56.101869 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.101787    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259f5008-f958-49b5-b794-5c525b55040a-maas-api-openshift-ca" pod "259f5008-f958-49b5-b794-5c525b55040a" (UID: "259f5008-f958-49b5-b794-5c525b55040a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:43:56.102578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.102554    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:56.102681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.102582    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259f5008-f958-49b5-b794-5c525b55040a-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:56.104195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.104168    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-xks-ca" pod "512964a3-c7ba-4bea-ab47-ab1566234737" (UID: "512964a3-c7ba-4bea-ab47-ab1566234737"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:56.104195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.104179    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512964a3-c7ba-4bea-ab47-ab1566234737-kube-api-access-b9b6s" pod "512964a3-c7ba-4bea-ab47-ab1566234737" (UID: "512964a3-c7ba-4bea-ab47-ab1566234737"). InnerVolumeSpecName "kube-api-access-b9b6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:56.104485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.104462    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259f5008-f958-49b5-b794-5c525b55040a-kube-api-access-9vcpp" pod "259f5008-f958-49b5-b794-5c525b55040a" (UID: "259f5008-f958-49b5-b794-5c525b55040a"). InnerVolumeSpecName "kube-api-access-9vcpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:56.104771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.104752    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/259f5008-f958-49b5-b794-5c525b55040a-maas-api-xks-ca" pod "259f5008-f958-49b5-b794-5c525b55040a" (UID: "259f5008-f958-49b5-b794-5c525b55040a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:56.204482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.204448    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vcpp\" (UniqueName: \"kubernetes.io/projected/259f5008-f958-49b5-b794-5c525b55040a-kube-api-access-9vcpp\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:56.204482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.204480    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259f5008-f958-49b5-b794-5c525b55040a-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:56.204702 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.204491    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9b6s\" (UniqueName: \"kubernetes.io/projected/512964a3-c7ba-4bea-ab47-ab1566234737-kube-api-access-b9b6s\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:56.204702 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.204501    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512964a3-c7ba-4bea-ab47-ab1566234737-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:43:56.827318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.820698    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq" event={"ID":"259f5008-f958-49b5-b794-5c525b55040a","Type":"ContainerDied","Data":"a5a42a032a5b3d4ae37cc2ef19a91fc96ed91ad70d696801b9b481fc927da66a"}
Nov 02 01:43:56.827318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.820804    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5a42a032a5b3d4ae37cc2ef19a91fc96ed91ad70d696801b9b481fc927da66a"
Nov 02 01:43:56.827318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.820897    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"
Nov 02 01:43:56.888317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.866016    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"
Nov 02 01:43:56.969318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.950601    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft" event={"ID":"512964a3-c7ba-4bea-ab47-ab1566234737","Type":"ContainerDied","Data":"0bcc820f05f178999cc24e029f4e51f6cc80eb44ddb2c973c1892bb0532c5eb8"}
Nov 02 01:43:56.984325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:56.984246    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bcc820f05f178999cc24e029f4e51f6cc80eb44ddb2c973c1892bb0532c5eb8"
Nov 02 01:43:57.625496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:57.625455    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p" podUID="14b4b0ad-6f18-410d-b545-db6f1ecd3141" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:43:57.625816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:57.625791    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h" podUID="5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:43:58.142315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:58.142110    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"]
Nov 02 01:43:58.195318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:58.193858    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ffe5c1-9dphn"]
Nov 02 01:43:58.372255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:58.372204    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"]
Nov 02 01:43:58.373121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:58.372881    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy" containerID="cri-o://ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32" gracePeriod=30
Nov 02 01:43:59.015717 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.013437    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87eda855-5122-4d86-87c6-a0fbebaa71a4" path="/var/lib/kubelet/pods/87eda855-5122-4d86-87c6-a0fbebaa71a4/volumes"
Nov 02 01:43:59.525034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.524990    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"]
Nov 02 01:43:59.525819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.525800    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5126a114-46de-4898-8088-c19a89fb51f1" containerName="istio-proxy"
Nov 02 01:43:59.525867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.525820    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5126a114-46de-4898-8088-c19a89fb51f1" containerName="istio-proxy"
Nov 02 01:43:59.525867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.525832    4851 cpu_manager.go:414] "containerMap: removing container" podUID="512964a3-c7ba-4bea-ab47-ab1566234737" containerName="revoke-keys"
Nov 02 01:43:59.525867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.525837    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512964a3-c7ba-4bea-ab47-ab1566234737" containerName="revoke-keys"
Nov 02 01:43:59.526030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.525876    4851 cpu_manager.go:414] "containerMap: removing container" podUID="259f5008-f958-49b5-b794-5c525b55040a" containerName="revoke-keys"
Nov 02 01:43:59.526030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.525880    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259f5008-f958-49b5-b794-5c525b55040a" containerName="revoke-keys"
Nov 02 01:43:59.526030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.525981    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5126a114-46de-4898-8088-c19a89fb51f1" containerName="istio-proxy"
Nov 02 01:43:59.526030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.525988    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="512964a3-c7ba-4bea-ab47-ab1566234737" containerName="revoke-keys"
Nov 02 01:43:59.526030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.525999    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="259f5008-f958-49b5-b794-5c525b55040a" containerName="revoke-keys"
Nov 02 01:43:59.532135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.532083    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.537076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.537052    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"]
Nov 02 01:43:59.537629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.537517    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-eaec8c-openshift-default-dockercfg-scjs9\""
Nov 02 01:43:59.620111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.620076    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-envoy\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.620310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.620141    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8656c6-8347-438b-b2ee-b2ae03c69590-istiod-ca-cert\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.620310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.620207    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-token\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.620430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.620349    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.620430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.620376    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-certs\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.620522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.620491    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlmpl\" (UniqueName: \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-kube-api-access-nlmpl\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.620522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.620512    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-credential-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.620617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.620540    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-data\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.620677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.620659    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-podinfo\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.731697    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmpl\" (UniqueName: \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-kube-api-access-nlmpl\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.731749    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-credential-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.731798    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-data\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.731871    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-podinfo\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.731966    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-envoy\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.732058    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8656c6-8347-438b-b2ee-b2ae03c69590-istiod-ca-cert\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.732171    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-token\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.732316    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.732345    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-certs\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.732928    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-certs\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.733597    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-credential-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.733829    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-data\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.740493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.738619    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.755332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.746195    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8656c6-8347-438b-b2ee-b2ae03c69590-istiod-ca-cert\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.778428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.771736    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlmpl\" (UniqueName: \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-kube-api-access-nlmpl\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.828314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.816815    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-token\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.829758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.828860    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-envoy\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.830496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.830470    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-podinfo\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:43:59.851976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:43:59.851946    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:44:00.334025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:00.333982    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"]
Nov 02 01:44:00.334674 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:00.334648    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8656c6_8347_438b_b2ee_b2ae03c69590.slice/crio-ea156564013fd572d4835e2fb4ccc354570c7db7382dab84214f40f2bb6efa67 WatchSource:0}: Error finding container ea156564013fd572d4835e2fb4ccc354570c7db7382dab84214f40f2bb6efa67: Status 404 returned error can't find the container with id ea156564013fd572d4835e2fb4ccc354570c7db7382dab84214f40f2bb6efa67
Nov 02 01:44:00.337440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:00.337404    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:00.337549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:00.337488    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:00.337629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:00.337581    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:01.066123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:01.066084    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" event={"ID":"fa8656c6-8347-438b-b2ee-b2ae03c69590","Type":"ContainerStarted","Data":"0511219424eac1bed532c7b2fe2bc593937ff78743c479f5b01dfe7d6a1a728a"}
Nov 02 01:44:01.066720 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:01.066142    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" event={"ID":"fa8656c6-8347-438b-b2ee-b2ae03c69590","Type":"ContainerStarted","Data":"ea156564013fd572d4835e2fb4ccc354570c7db7382dab84214f40f2bb6efa67"}
Nov 02 01:44:01.093699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:01.093642    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" podStartSLOduration=2.093628712 podStartE2EDuration="2.093628712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:01.089480857 +0000 UTC m=+1746.394413685" watchObservedRunningTime="2026-11-02 01:44:01.093628712 +0000 UTC m=+1746.398561537"
Nov 02 01:44:01.295102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:01.295067    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"]
Nov 02 01:44:01.300226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:01.300199    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9a4595-8qtdz"]
Nov 02 01:44:01.853582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:01.853547    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:44:01.859790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:01.859761    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:44:01.956042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:01.956009    4851 scope.go:122] "RemoveContainer" containerID="8c84a22d516eb203c97ff0791d2743487f5249a6679deee1c6fcf44651a8547c"
Nov 02 01:44:01.970219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:01.970192    4851 scope.go:122] "RemoveContainer" containerID="f1f21637f77bc57322efb9d5b774991b26b35ac9202db396b7042ac74c715e21"
Nov 02 01:44:02.080896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.080873    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:44:02.081947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.081897    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:44:02.103759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.103696    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"]
Nov 02 01:44:02.110072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.110040    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c22a0d-pzz2z"]
Nov 02 01:44:02.760905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.760876    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:44:02.811373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.811346    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:44:02.865649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.865135    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-plugins-config-volume\") pod \"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68\" (UID: \"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68\") "
Nov 02 01:44:02.865649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.865195    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-kube-api-access-whcwf\" (UniqueName: \"kubernetes.io/projected/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-kube-api-access-whcwf\") pod \"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68\" (UID: \"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68\") "
Nov 02 01:44:02.865649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.865489    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c959c5af-7076-4638-8878-ef9f20ec5902" path="/var/lib/kubelet/pods/c959c5af-7076-4638-8878-ef9f20ec5902/volumes"
Nov 02 01:44:02.866181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.866154    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-plugins-config-volume" pod "5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" (UID: "5ffbeda9-0b12-49cc-b5c6-6922c4bacf68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:02.866452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.866425    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d267cca5-7ae8-452a-bf78-dc59b87bf208" path="/var/lib/kubelet/pods/d267cca5-7ae8-452a-bf78-dc59b87bf208/volumes"
Nov 02 01:44:02.866630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.866607    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.868962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.868933    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-kube-api-access-whcwf" pod "5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" (UID: "5ffbeda9-0b12-49cc-b5c6-6922c4bacf68"). InnerVolumeSpecName "kube-api-access-whcwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:02.967967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.967936    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b4b0ad-6f18-410d-b545-db6f1ecd3141-kube-api-access-4nvl7\" (UniqueName: \"kubernetes.io/projected/14b4b0ad-6f18-410d-b545-db6f1ecd3141-kube-api-access-4nvl7\") pod \"14b4b0ad-6f18-410d-b545-db6f1ecd3141\" (UID: \"14b4b0ad-6f18-410d-b545-db6f1ecd3141\") "
Nov 02 01:44:02.968158 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.968104    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b4b0ad-6f18-410d-b545-db6f1ecd3141-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b4b0ad-6f18-410d-b545-db6f1ecd3141-plugins-config-volume\") pod \"14b4b0ad-6f18-410d-b545-db6f1ecd3141\" (UID: \"14b4b0ad-6f18-410d-b545-db6f1ecd3141\") "
Nov 02 01:44:02.968456 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.968433    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b4b0ad-6f18-410d-b545-db6f1ecd3141-plugins-config-volume" pod "14b4b0ad-6f18-410d-b545-db6f1ecd3141" (UID: "14b4b0ad-6f18-410d-b545-db6f1ecd3141"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:02.968945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.968883    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b4b0ad-6f18-410d-b545-db6f1ecd3141-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.968945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.968912    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whcwf\" (UniqueName: \"kubernetes.io/projected/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68-kube-api-access-whcwf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.970253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:02.970229    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b4b0ad-6f18-410d-b545-db6f1ecd3141-kube-api-access-4nvl7" pod "14b4b0ad-6f18-410d-b545-db6f1ecd3141" (UID: "14b4b0ad-6f18-410d-b545-db6f1ecd3141"). InnerVolumeSpecName "kube-api-access-4nvl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:03.070935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.070895    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvl7\" (UniqueName: \"kubernetes.io/projected/14b4b0ad-6f18-410d-b545-db6f1ecd3141-kube-api-access-4nvl7\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:03.089772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.089741    4851 generic.go:356] "Generic (PLEG): container finished" podID="5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" containerID="82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c" exitCode=0
Nov 02 01:44:03.090272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.089805    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"
Nov 02 01:44:03.090272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.089821    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h" event={"ID":"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68","Type":"ContainerDied","Data":"82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c"}
Nov 02 01:44:03.090272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.089870    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h" event={"ID":"5ffbeda9-0b12-49cc-b5c6-6922c4bacf68","Type":"ContainerDied","Data":"0a4fe3b27bc60aef3a4e99cb28d4d6c8f7f4b9b9c88e88ccf656add2af16919f"}
Nov 02 01:44:03.090272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.089903    4851 scope.go:122] "RemoveContainer" containerID="82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c"
Nov 02 01:44:03.091536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.091514    4851 generic.go:356] "Generic (PLEG): container finished" podID="14b4b0ad-6f18-410d-b545-db6f1ecd3141" containerID="e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba" exitCode=0
Nov 02 01:44:03.091614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.091569    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"
Nov 02 01:44:03.091614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.091596    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p" event={"ID":"14b4b0ad-6f18-410d-b545-db6f1ecd3141","Type":"ContainerDied","Data":"e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba"}
Nov 02 01:44:03.091726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.091645    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p" event={"ID":"14b4b0ad-6f18-410d-b545-db6f1ecd3141","Type":"ContainerDied","Data":"41c603172a7b06bd1af49163852eb9a38f8835c843f60c835fb07523f118df4e"}
Nov 02 01:44:03.103936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.103918    4851 scope.go:122] "RemoveContainer" containerID="82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c"
Nov 02 01:44:03.104208 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:03.104186    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c\": container with ID starting with 82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c not found: ID does not exist" containerID="82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c"
Nov 02 01:44:03.104353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.104220    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c"} err="failed to get container status \"82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c\": rpc error: code = NotFound desc = could not find container \"82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c\": container with ID starting with 82d4681153a1d4d736af6f7f43415393e87e9cb4cadfe029993110ebab5dce5c not found: ID does not exist"
Nov 02 01:44:03.104353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.104244    4851 scope.go:122] "RemoveContainer" containerID="e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba"
Nov 02 01:44:03.115117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.115085    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"]
Nov 02 01:44:03.118015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.117995    4851 scope.go:122] "RemoveContainer" containerID="e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba"
Nov 02 01:44:03.118422 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:03.118394    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba\": container with ID starting with e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba not found: ID does not exist" containerID="e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba"
Nov 02 01:44:03.118489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.118435    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba"} err="failed to get container status \"e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba\": rpc error: code = NotFound desc = could not find container \"e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba\": container with ID starting with e7f1e19be5003544dd78f624fa085db188a2458db4220707731be9044bb691ba not found: ID does not exist"
Nov 02 01:44:03.120473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.120455    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-47dd19-f47c59d5-qnj5h"]
Nov 02 01:44:03.131519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.131491    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"]
Nov 02 01:44:03.135484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.135459    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-47dd19-7c4c948447-5265p"]
Nov 02 01:44:03.805475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.805451    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:44:03.883712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.883676    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-token\" (UniqueName: \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-token\") pod \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") "
Nov 02 01:44:03.883918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.883732    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istiod-ca-cert\") pod \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") "
Nov 02 01:44:03.883918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.883774    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-data\") pod \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") "
Nov 02 01:44:03.883918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.883891    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-credential-socket\") pod \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") "
Nov 02 01:44:03.884105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.884085    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-data" pod "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" (UID: "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:03.884168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.884148    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-envoy\") pod \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") "
Nov 02 01:44:03.884233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.884174    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-socket\") pod \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") "
Nov 02 01:44:03.884233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.884219    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-certs\") pod \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") "
Nov 02 01:44:03.884394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.884250    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-podinfo\") pod \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") "
Nov 02 01:44:03.884394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.884306    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-kube-api-access-4zw87\" (UniqueName: \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-kube-api-access-4zw87\") pod \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\" (UID: \"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa\") "
Nov 02 01:44:03.884394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.884303    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-credential-socket" pod "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" (UID: "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:03.884598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.884564    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-certs" pod "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" (UID: "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:03.884957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.884799    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istiod-ca-cert" pod "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" (UID: "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:03.884957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.884836    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-socket" pod "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" (UID: "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:03.885209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.885176    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:03.885209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.885202    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:03.885434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.885217    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:03.885434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.885232    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:03.885434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.885247    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:03.887465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.887439    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-envoy" pod "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" (UID: "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:03.887698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.887662    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-token" pod "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" (UID: "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:03.888960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.888934    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-kube-api-access-4zw87" pod "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" (UID: "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa"). InnerVolumeSpecName "kube-api-access-4zw87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:03.889070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.888974    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-podinfo" pod "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" (UID: "d9d3bd93-9b1d-492c-bfa6-6d62b29513fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:44:03.973852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.973769    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns" podUID="4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:44:03.987038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.987005    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:03.987038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.987040    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:03.987245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.987055    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zw87\" (UniqueName: \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-kube-api-access-4zw87\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:03.987245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:03.987072    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:04.102912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.102875    4851 generic.go:356] "Generic (PLEG): container finished" podID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerID="ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32" exitCode=0
Nov 02 01:44:04.103447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.102947    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"
Nov 02 01:44:04.103447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.102958    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" event={"ID":"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa","Type":"ContainerDied","Data":"ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32"}
Nov 02 01:44:04.103447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.103021    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq" event={"ID":"d9d3bd93-9b1d-492c-bfa6-6d62b29513fa","Type":"ContainerDied","Data":"69ce08fb9345414c5846d711e5700f6e106e385d1cc6e6df16e19014a24425a2"}
Nov 02 01:44:04.103447 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.103051    4851 scope.go:122] "RemoveContainer" containerID="ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32"
Nov 02 01:44:04.116264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.116247    4851 scope.go:122] "RemoveContainer" containerID="2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41"
Nov 02 01:44:04.127609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.127588    4851 scope.go:122] "RemoveContainer" containerID="ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32"
Nov 02 01:44:04.127921 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:04.127898    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32\": container with ID starting with ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32 not found: ID does not exist" containerID="ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32"
Nov 02 01:44:04.127997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.127929    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32"} err="failed to get container status \"ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32\": rpc error: code = NotFound desc = could not find container \"ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32\": container with ID starting with ca0585ab9f75435674ae8bfb0cfab9666b77e746dfe7b733af40955bfb177f32 not found: ID does not exist"
Nov 02 01:44:04.127997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.127946    4851 scope.go:122] "RemoveContainer" containerID="2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41"
Nov 02 01:44:04.128278 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:04.128254    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41\": container with ID starting with 2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41 not found: ID does not exist" containerID="2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41"
Nov 02 01:44:04.128413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.128326    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41"} err="failed to get container status \"2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41\": rpc error: code = NotFound desc = could not find container \"2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41\": container with ID starting with 2eba7caa5c27880ec0687c0ee3213df8832d55e1fb9f9ee49612e5eb8df87f41 not found: ID does not exist"
Nov 02 01:44:04.130760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.130736    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"]
Nov 02 01:44:04.134677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.134653    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-c8kkq"]
Nov 02 01:44:04.851685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.851653    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b4b0ad-6f18-410d-b545-db6f1ecd3141" path="/var/lib/kubelet/pods/14b4b0ad-6f18-410d-b545-db6f1ecd3141/volumes"
Nov 02 01:44:04.852180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.852159    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" path="/var/lib/kubelet/pods/5ffbeda9-0b12-49cc-b5c6-6922c4bacf68/volumes"
Nov 02 01:44:04.852695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.852675    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" path="/var/lib/kubelet/pods/d9d3bd93-9b1d-492c-bfa6-6d62b29513fa/volumes"
Nov 02 01:44:04.987896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.987865    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"]
Nov 02 01:44:04.988677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.988657    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy"
Nov 02 01:44:04.988677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.988677    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy"
Nov 02 01:44:04.988865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.988706    4851 cpu_manager.go:414] "containerMap: removing container" podUID="14b4b0ad-6f18-410d-b545-db6f1ecd3141" containerName="payload-pre-processing"
Nov 02 01:44:04.988865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.988714    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b4b0ad-6f18-410d-b545-db6f1ecd3141" containerName="payload-pre-processing"
Nov 02 01:44:04.988865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.988746    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" containerName="payload-processing"
Nov 02 01:44:04.988865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.988753    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" containerName="payload-processing"
Nov 02 01:44:04.989133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.988948    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy"
Nov 02 01:44:04.989133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.988967    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b4b0ad-6f18-410d-b545-db6f1ecd3141" containerName="payload-pre-processing"
Nov 02 01:44:04.989133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.988982    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffbeda9-0b12-49cc-b5c6-6922c4bacf68" containerName="payload-processing"
Nov 02 01:44:04.994580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:04.994520    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:44:05.003278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.003256    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-eaec8c-dockercfg-t52rx\""
Nov 02 01:44:05.004483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.004458    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-eaec8c\""
Nov 02 01:44:05.004612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.004523    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"]
Nov 02 01:44:05.029893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.029841    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"]
Nov 02 01:44:05.030614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.030594    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy"
Nov 02 01:44:05.030614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.030611    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy"
Nov 02 01:44:05.030822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.030726    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d3bd93-9b1d-492c-bfa6-6d62b29513fa" containerName="istio-proxy"
Nov 02 01:44:05.034565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.034543    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:44:05.039376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.039352    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"]
Nov 02 01:44:05.141386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.119675    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7crc\" (UniqueName: \"kubernetes.io/projected/31addff9-dc30-4d8a-8753-8229f834003f-kube-api-access-j7crc\") pod \"payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g\" (UID: \"31addff9-dc30-4d8a-8753-8229f834003f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:44:05.141386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.119858    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77229192-c71d-4a53-becc-3338c7de9c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57\" (UID: \"77229192-c71d-4a53-becc-3338c7de9c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:44:05.141386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.119955    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x6q4\" (UniqueName: \"kubernetes.io/projected/77229192-c71d-4a53-becc-3338c7de9c81-kube-api-access-7x6q4\") pod \"payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57\" (UID: \"77229192-c71d-4a53-becc-3338c7de9c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:44:05.141386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.120140    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31addff9-dc30-4d8a-8753-8229f834003f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g\" (UID: \"31addff9-dc30-4d8a-8753-8229f834003f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:44:05.271191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.271149    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7crc\" (UniqueName: \"kubernetes.io/projected/31addff9-dc30-4d8a-8753-8229f834003f-kube-api-access-j7crc\") pod \"payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g\" (UID: \"31addff9-dc30-4d8a-8753-8229f834003f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:44:05.271416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.271346    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77229192-c71d-4a53-becc-3338c7de9c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57\" (UID: \"77229192-c71d-4a53-becc-3338c7de9c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:44:05.271416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.271407    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x6q4\" (UniqueName: \"kubernetes.io/projected/77229192-c71d-4a53-becc-3338c7de9c81-kube-api-access-7x6q4\") pod \"payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57\" (UID: \"77229192-c71d-4a53-becc-3338c7de9c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:44:05.271591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.271568    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31addff9-dc30-4d8a-8753-8229f834003f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g\" (UID: \"31addff9-dc30-4d8a-8753-8229f834003f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:44:05.272584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.272560    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77229192-c71d-4a53-becc-3338c7de9c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57\" (UID: \"77229192-c71d-4a53-becc-3338c7de9c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:44:05.272733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.272611    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31addff9-dc30-4d8a-8753-8229f834003f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g\" (UID: \"31addff9-dc30-4d8a-8753-8229f834003f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:44:05.321314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.314872    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7crc\" (UniqueName: \"kubernetes.io/projected/31addff9-dc30-4d8a-8753-8229f834003f-kube-api-access-j7crc\") pod \"payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g\" (UID: \"31addff9-dc30-4d8a-8753-8229f834003f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:44:05.338315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.332967    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x6q4\" (UniqueName: \"kubernetes.io/projected/77229192-c71d-4a53-becc-3338c7de9c81-kube-api-access-7x6q4\") pod \"payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57\" (UID: \"77229192-c71d-4a53-becc-3338c7de9c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:44:05.355341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.352798    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:44:05.613493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.611742    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:44:05.728657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.728622    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"]
Nov 02 01:44:05.728913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.728881    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerName="istio-proxy" containerID="cri-o://9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850" gracePeriod=30
Nov 02 01:44:05.911756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:05.911662    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"]
Nov 02 01:44:05.970243 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:05.970179    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77229192_c71d_4a53_becc_3338c7de9c81.slice/crio-a1bf7ae02431693f1bc1489f5add333cc72610971200d52ab8c48742c505cbab WatchSource:0}: Error finding container a1bf7ae02431693f1bc1489f5add333cc72610971200d52ab8c48742c505cbab: Status 404 returned error can't find the container with id a1bf7ae02431693f1bc1489f5add333cc72610971200d52ab8c48742c505cbab
Nov 02 01:44:06.344983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.344957    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"]
Nov 02 01:44:06.345467 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:06.345444    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31addff9_dc30_4d8a_8753_8229f834003f.slice/crio-537cd0fd52ef78edf374d8191c363998cbfa4e7b17c3240a75f44ca7e7408448 WatchSource:0}: Error finding container 537cd0fd52ef78edf374d8191c363998cbfa4e7b17c3240a75f44ca7e7408448: Status 404 returned error can't find the container with id 537cd0fd52ef78edf374d8191c363998cbfa4e7b17c3240a75f44ca7e7408448
Nov 02 01:44:06.391892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.391865    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g" event={"ID":"31addff9-dc30-4d8a-8753-8229f834003f","Type":"ContainerStarted","Data":"537cd0fd52ef78edf374d8191c363998cbfa4e7b17c3240a75f44ca7e7408448"}
Nov 02 01:44:06.393751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.393723    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57" event={"ID":"77229192-c71d-4a53-becc-3338c7de9c81","Type":"ContainerStarted","Data":"a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b"}
Nov 02 01:44:06.393858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.393778    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57" event={"ID":"77229192-c71d-4a53-becc-3338c7de9c81","Type":"ContainerStarted","Data":"a1bf7ae02431693f1bc1489f5add333cc72610971200d52ab8c48742c505cbab"}
Nov 02 01:44:06.393858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.393818    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:44:06.416130 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.416092    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57" podStartSLOduration=2.416078377 podStartE2EDuration="2.416078377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:06.412560308 +0000 UTC m=+1751.717493135" watchObservedRunningTime="2026-11-02 01:44:06.416078377 +0000 UTC m=+1751.721011246"
Nov 02 01:44:06.786309 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.786263    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"]
Nov 02 01:44:06.790951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.790931    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:06.799345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.799319    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:44:06.799458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.799432    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:44:06.799676 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.799635    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:44:06.800376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.800355    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:44:06.803330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.803308    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"]
Nov 02 01:44:06.907592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.907560    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"]
Nov 02 01:44:06.911019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.910999    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95671cc9-mdxbs\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:06.911173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.911154    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95671cc9-mdxbs\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:06.911386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.911369    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlzx\" (UniqueName: \"kubernetes.io/projected/bd00767b-a324-4d8c-96a5-30fee3f05008-kube-api-access-9xlzx\") pod \"maas-api-revoke-sub-95671cc9-mdxbs\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:06.912233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.912218    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:06.934275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.934250    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"]
Nov 02 01:44:06.964528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:06.964484    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-3c4ea5-openshift-default-dockercfg-cm4v8\""
Nov 02 01:44:07.013072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013045    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-token\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.013248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013078    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-data\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.013248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013198    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-envoy\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.013406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013249    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlzx\" (UniqueName: \"kubernetes.io/projected/bd00767b-a324-4d8c-96a5-30fee3f05008-kube-api-access-9xlzx\") pod \"maas-api-revoke-sub-95671cc9-mdxbs\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:07.013406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013349    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h4q2\" (UniqueName: \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-kube-api-access-5h4q2\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.013527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013502    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-certs\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.013585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013537    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-podinfo\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.013585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013563    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95671cc9-mdxbs\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:07.013730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013713    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-socket\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.013888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013861    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95671cc9-mdxbs\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:07.014026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013910    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-credential-socket\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.014026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.013949    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5377e2b2-cc37-4884-96a0-9a5e2628d968-istiod-ca-cert\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.014174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.014153    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95671cc9-mdxbs\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:07.016401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.016383    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95671cc9-mdxbs\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:07.041595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.041542    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlzx\" (UniqueName: \"kubernetes.io/projected/bd00767b-a324-4d8c-96a5-30fee3f05008-kube-api-access-9xlzx\") pod \"maas-api-revoke-sub-95671cc9-mdxbs\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:07.108609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.108578    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:07.115820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.115791    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-data\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.115962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.115883    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-envoy\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.115962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.115924    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h4q2\" (UniqueName: \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-kube-api-access-5h4q2\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.115970    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-certs\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.115993    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-podinfo\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.116044    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-socket\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.116141    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-credential-socket\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.116169    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5377e2b2-cc37-4884-96a0-9a5e2628d968-istiod-ca-cert\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.116251    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-token\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.116551    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-socket\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.116597    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-credential-socket\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.116632    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-certs\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.116792    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-data\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.116946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.116925    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5377e2b2-cc37-4884-96a0-9a5e2628d968-istiod-ca-cert\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.118762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.118741    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-envoy\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.118998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.118980    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-podinfo\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.152677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.152648    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h4q2\" (UniqueName: \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-kube-api-access-5h4q2\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.152985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.152964    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-token\") pod \"e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") " pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.227324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.227275    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:07.588369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.588027    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"]
Nov 02 01:44:07.602058 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.595481    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g" event={"ID":"31addff9-dc30-4d8a-8753-8229f834003f","Type":"ContainerStarted","Data":"e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65"}
Nov 02 01:44:07.602058 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.595673    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:44:07.633642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.633594    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g" podStartSLOduration=3.633579799 podStartE2EDuration="3.633579799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:07.627866952 +0000 UTC m=+1752.932799850" watchObservedRunningTime="2026-11-02 01:44:07.633579799 +0000 UTC m=+1752.938512661"
Nov 02 01:44:07.643021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.642989    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"]
Nov 02 01:44:07.645528 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:07.645498    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5377e2b2_cc37_4884_96a0_9a5e2628d968.slice/crio-ee66fc30639e44a22230172ecb5e56e104539f93ab075a2163231daffaf40d35 WatchSource:0}: Error finding container ee66fc30639e44a22230172ecb5e56e104539f93ab075a2163231daffaf40d35: Status 404 returned error can't find the container with id ee66fc30639e44a22230172ecb5e56e104539f93ab075a2163231daffaf40d35
Nov 02 01:44:07.661017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.660971    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:07.661144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.661066    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:07.661144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:07.661111    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:08.614621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:08.614582    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" event={"ID":"5377e2b2-cc37-4884-96a0-9a5e2628d968","Type":"ContainerStarted","Data":"425c1575a0d8a761f7b21ee2d9b69e7b07dddbcc0c7656a5454786f8eab9141f"}
Nov 02 01:44:08.615095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:08.614695    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" event={"ID":"5377e2b2-cc37-4884-96a0-9a5e2628d968","Type":"ContainerStarted","Data":"ee66fc30639e44a22230172ecb5e56e104539f93ab075a2163231daffaf40d35"}
Nov 02 01:44:08.616542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:08.616514    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs" event={"ID":"bd00767b-a324-4d8c-96a5-30fee3f05008","Type":"ContainerStarted","Data":"a8762c622475e758b92e0a854472d0eb8b9267717357ce0cfe2ecfd30a17bdee"}
Nov 02 01:44:08.616652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:08.616561    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs" event={"ID":"bd00767b-a324-4d8c-96a5-30fee3f05008","Type":"ContainerStarted","Data":"96b1da04266dc8561414d5c6c00c520029548e5b4dca304ea58ab1c250861555"}
Nov 02 01:44:08.639389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:08.639336    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" podStartSLOduration=2.639319207 podStartE2EDuration="2.639319207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:08.637342832 +0000 UTC m=+1753.942275660" watchObservedRunningTime="2026-11-02 01:44:08.639319207 +0000 UTC m=+1753.944252034"
Nov 02 01:44:08.657917 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:08.657866    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs" podStartSLOduration=2.657850368 podStartE2EDuration="2.657850368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:08.656253033 +0000 UTC m=+1753.961185863" watchObservedRunningTime="2026-11-02 01:44:08.657850368 +0000 UTC m=+1753.962783191"
Nov 02 01:44:09.232665 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:09.229731    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:09.264316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:09.247515    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:09.671120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:09.669768    4851 generic.go:356] "Generic (PLEG): container finished" podID="bd00767b-a324-4d8c-96a5-30fee3f05008" containerID="a8762c622475e758b92e0a854472d0eb8b9267717357ce0cfe2ecfd30a17bdee" exitCode=0
Nov 02 01:44:09.671120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:09.670800    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs" event={"ID":"bd00767b-a324-4d8c-96a5-30fee3f05008","Type":"ContainerDied","Data":"a8762c622475e758b92e0a854472d0eb8b9267717357ce0cfe2ecfd30a17bdee"}
Nov 02 01:44:09.671120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:09.670869    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:09.672362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:09.672283    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:44:11.425335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.420853    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"]
Nov 02 01:44:11.425335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.423972    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy" containerID="cri-o://2b681c18b956eaa5c0a9db4063765ff4564890e1b9e1ae9726686a31d891e5df" gracePeriod=30
Nov 02 01:44:11.518008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.517979    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:11.643050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.643025    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:44:11.656537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.656507    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"]
Nov 02 01:44:11.657349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.657326    4851 cpu_manager.go:414] "containerMap: removing container" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerName="istio-proxy"
Nov 02 01:44:11.657349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.657347    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerName="istio-proxy"
Nov 02 01:44:11.657573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.657375    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bd00767b-a324-4d8c-96a5-30fee3f05008" containerName="revoke-keys"
Nov 02 01:44:11.657573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.657384    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd00767b-a324-4d8c-96a5-30fee3f05008" containerName="revoke-keys"
Nov 02 01:44:11.657573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.657537    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerName="istio-proxy"
Nov 02 01:44:11.657573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.657550    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd00767b-a324-4d8c-96a5-30fee3f05008" containerName="revoke-keys"
Nov 02 01:44:11.661133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.661105    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:11.668517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.668401    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd00767b-a324-4d8c-96a5-30fee3f05008-kube-api-access-9xlzx\" (UniqueName: \"kubernetes.io/projected/bd00767b-a324-4d8c-96a5-30fee3f05008-kube-api-access-9xlzx\") pod \"bd00767b-a324-4d8c-96a5-30fee3f05008\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") "
Nov 02 01:44:11.668517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.668513    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-xks-ca\") pod \"bd00767b-a324-4d8c-96a5-30fee3f05008\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") "
Nov 02 01:44:11.668860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.668840    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-openshift-ca\") pod \"bd00767b-a324-4d8c-96a5-30fee3f05008\" (UID: \"bd00767b-a324-4d8c-96a5-30fee3f05008\") "
Nov 02 01:44:11.669429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.669406    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-openshift-ca" pod "bd00767b-a324-4d8c-96a5-30fee3f05008" (UID: "bd00767b-a324-4d8c-96a5-30fee3f05008"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:11.671506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.671481    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:11.673883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.673359    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gznqd\" (UniqueName: \"kubernetes.io/projected/9b1f6375-0a3f-4053-8398-289a711be8ac-kube-api-access-gznqd\") pod \"maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:11.673883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.673450    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:11.673883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.673869    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.674508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.674236    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd00767b-a324-4d8c-96a5-30fee3f05008-kube-api-access-9xlzx" pod "bd00767b-a324-4d8c-96a5-30fee3f05008" (UID: "bd00767b-a324-4d8c-96a5-30fee3f05008"). InnerVolumeSpecName "kube-api-access-9xlzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:11.682349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.681726    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"]
Nov 02 01:44:11.684894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.684861    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-xks-ca" pod "bd00767b-a324-4d8c-96a5-30fee3f05008" (UID: "bd00767b-a324-4d8c-96a5-30fee3f05008"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:44:11.703523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.703491    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs" event={"ID":"bd00767b-a324-4d8c-96a5-30fee3f05008","Type":"ContainerDied","Data":"96b1da04266dc8561414d5c6c00c520029548e5b4dca304ea58ab1c250861555"}
Nov 02 01:44:11.703722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.703539    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96b1da04266dc8561414d5c6c00c520029548e5b4dca304ea58ab1c250861555"
Nov 02 01:44:11.703722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.703594    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"
Nov 02 01:44:11.707656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.707622    4851 generic.go:356] "Generic (PLEG): container finished" podID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" containerID="9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850" exitCode=0
Nov 02 01:44:11.707809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.707681    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"
Nov 02 01:44:11.707809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.707752    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" event={"ID":"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2","Type":"ContainerDied","Data":"9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850"}
Nov 02 01:44:11.707942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.707809    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2" event={"ID":"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2","Type":"ContainerDied","Data":"5cb4aca257aae58f04f3635fd85a8ac58ef952453621e1d8cc3023ecb693a0a6"}
Nov 02 01:44:11.707942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.707839    4851 scope.go:122] "RemoveContainer" containerID="9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850"
Nov 02 01:44:11.736656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.736359    4851 scope.go:122] "RemoveContainer" containerID="9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850"
Nov 02 01:44:11.736829 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:11.736784    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850\": container with ID starting with 9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850 not found: ID does not exist" containerID="9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850"
Nov 02 01:44:11.736929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.736843    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850"} err="failed to get container status \"9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850\": rpc error: code = NotFound desc = could not find container \"9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850\": container with ID starting with 9b38fd0834b99da0035d140a25b90c9a9094789b8c4a42789dcc420d5ca58850 not found: ID does not exist"
Nov 02 01:44:11.774871 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.774839    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-token\" (UniqueName: \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-token\") pod \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") "
Nov 02 01:44:11.774995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.774911    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-socket\") pod \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") "
Nov 02 01:44:11.774995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.774972    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istiod-ca-cert\") pod \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") "
Nov 02 01:44:11.775090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775006    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-certs\") pod \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") "
Nov 02 01:44:11.775158 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775126    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-envoy\") pod \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") "
Nov 02 01:44:11.776044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775154    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-kube-api-access-2x669\" (UniqueName: \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-kube-api-access-2x669\") pod \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") "
Nov 02 01:44:11.776044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775317    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-credential-socket\") pod \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") "
Nov 02 01:44:11.776044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775371    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-podinfo\") pod \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") "
Nov 02 01:44:11.776044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775403    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-data\") pod \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\" (UID: \"f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2\") "
Nov 02 01:44:11.776044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775450    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-socket" pod "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" (UID: "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:11.776044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775462    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-certs" pod "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" (UID: "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:11.776044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775615    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-credential-socket" pod "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" (UID: "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:11.776044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775619    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gznqd\" (UniqueName: \"kubernetes.io/projected/9b1f6375-0a3f-4053-8398-289a711be8ac-kube-api-access-gznqd\") pod \"maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:11.776044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775727    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istiod-ca-cert" pod "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" (UID: "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:11.776044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.775742    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:11.776503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.776076    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:11.776503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.776178    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.776503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.776197    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.776503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.776213    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.776503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.776228    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.776503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.776242    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xlzx\" (UniqueName: \"kubernetes.io/projected/bd00767b-a324-4d8c-96a5-30fee3f05008-kube-api-access-9xlzx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.776503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.776257    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd00767b-a324-4d8c-96a5-30fee3f05008-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.776838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.776692    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:11.777188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.777036    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-data" pod "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" (UID: "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:11.786704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.780442    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-podinfo" pod "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" (UID: "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:44:11.786704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.784110    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:11.786704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.786065    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-kube-api-access-2x669" pod "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" (UID: "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2"). InnerVolumeSpecName "kube-api-access-2x669". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:11.788001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.787971    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-token" pod "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" (UID: "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:11.788329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.788276    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-envoy" pod "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" (UID: "f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:11.789888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.789863    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gznqd\" (UniqueName: \"kubernetes.io/projected/9b1f6375-0a3f-4053-8398-289a711be8ac-kube-api-access-gznqd\") pod \"maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:11.877947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.877914    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.877947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.877947    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.878164 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.877961    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x669\" (UniqueName: \"kubernetes.io/projected/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-kube-api-access-2x669\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.878164 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.877974    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.878164 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.877986    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:11.987061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:11.986968    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:12.076315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.074950    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"]
Nov 02 01:44:12.101555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.101518    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-kcvv2"]
Nov 02 01:44:12.285351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.285319    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"]
Nov 02 01:44:12.285664 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:12.285639    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1f6375_0a3f_4053_8398_289a711be8ac.slice/crio-f6ed21188ef80f7665946dae20f76bbee0df743ed2d1bda3ed9b34f6490d33e4 WatchSource:0}: Error finding container f6ed21188ef80f7665946dae20f76bbee0df743ed2d1bda3ed9b34f6490d33e4: Status 404 returned error can't find the container with id f6ed21188ef80f7665946dae20f76bbee0df743ed2d1bda3ed9b34f6490d33e4
Nov 02 01:44:12.724895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.724842    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m" event={"ID":"9b1f6375-0a3f-4053-8398-289a711be8ac","Type":"ContainerStarted","Data":"686e7f34449c4cfaee7367d23925e7bc99c59c8165fc718fefaef9dce7cf642e"}
Nov 02 01:44:12.725461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.724915    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m" event={"ID":"9b1f6375-0a3f-4053-8398-289a711be8ac","Type":"ContainerStarted","Data":"f6ed21188ef80f7665946dae20f76bbee0df743ed2d1bda3ed9b34f6490d33e4"}
Nov 02 01:44:12.745399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.745337    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m" podStartSLOduration=1.745319117 podStartE2EDuration="1.745319117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:12.744252081 +0000 UTC m=+1758.049184908" watchObservedRunningTime="2026-11-02 01:44:12.745319117 +0000 UTC m=+1758.050251921"
Nov 02 01:44:12.776928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.776894    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"]
Nov 02 01:44:12.781696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.781668    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:44:12.788926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.786130    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-3c4ea5\""
Nov 02 01:44:12.795746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.794268    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"]
Nov 02 01:44:12.803576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.803530    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm\" (UID: \"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:44:12.803855 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.803838    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h92q\" (UniqueName: \"kubernetes.io/projected/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-kube-api-access-5h92q\") pod \"payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm\" (UID: \"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:44:12.811979 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.811940    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"]
Nov 02 01:44:12.816740 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.816711    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:44:12.823453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.823422    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-3c4ea5-dockercfg-59m4p\""
Nov 02 01:44:12.825685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.825652    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"]
Nov 02 01:44:12.850494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.850462    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2" path="/var/lib/kubelet/pods/f2eac2ec-6b60-4d8e-a0bf-ff607cb68de2/volumes"
Nov 02 01:44:12.943404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.939860    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tg6g\" (UniqueName: \"kubernetes.io/projected/49ae4474-4f10-47b8-86a3-d87b2f6e6021-kube-api-access-9tg6g\") pod \"payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq\" (UID: \"49ae4474-4f10-47b8-86a3-d87b2f6e6021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:44:12.970472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.970427    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ae4474-4f10-47b8-86a3-d87b2f6e6021-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq\" (UID: \"49ae4474-4f10-47b8-86a3-d87b2f6e6021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:44:12.970934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.970905    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h92q\" (UniqueName: \"kubernetes.io/projected/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-kube-api-access-5h92q\") pod \"payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm\" (UID: \"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:44:12.971326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:12.971283    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm\" (UID: \"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:44:13.008845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.008812    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm\" (UID: \"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:44:13.077491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.074048    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tg6g\" (UniqueName: \"kubernetes.io/projected/49ae4474-4f10-47b8-86a3-d87b2f6e6021-kube-api-access-9tg6g\") pod \"payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq\" (UID: \"49ae4474-4f10-47b8-86a3-d87b2f6e6021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:44:13.077491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.074126    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ae4474-4f10-47b8-86a3-d87b2f6e6021-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq\" (UID: \"49ae4474-4f10-47b8-86a3-d87b2f6e6021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:44:13.077491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.075268    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ae4474-4f10-47b8-86a3-d87b2f6e6021-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq\" (UID: \"49ae4474-4f10-47b8-86a3-d87b2f6e6021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:44:13.087617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.080595    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h92q\" (UniqueName: \"kubernetes.io/projected/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-kube-api-access-5h92q\") pod \"payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm\" (UID: \"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:44:13.096980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.096945    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tg6g\" (UniqueName: \"kubernetes.io/projected/49ae4474-4f10-47b8-86a3-d87b2f6e6021-kube-api-access-9tg6g\") pod \"payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq\" (UID: \"49ae4474-4f10-47b8-86a3-d87b2f6e6021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:44:13.123652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.123381    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:44:13.147318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.144709    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:44:13.570405 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:13.570368    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a2b0ca_9b50_4b68_b0bb_7d7425bd3655.slice/crio-724709d1d37caf0fa31958b9446a70941e22a7cfd83cfa6549509b73195d2c10 WatchSource:0}: Error finding container 724709d1d37caf0fa31958b9446a70941e22a7cfd83cfa6549509b73195d2c10: Status 404 returned error can't find the container with id 724709d1d37caf0fa31958b9446a70941e22a7cfd83cfa6549509b73195d2c10
Nov 02 01:44:13.571578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.571554    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"]
Nov 02 01:44:13.573415 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:13.573390    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ae4474_4f10_47b8_86a3_d87b2f6e6021.slice/crio-e8b90d47c151794823be869769ce846598c98c430d39f395f24a7f4da4856989 WatchSource:0}: Error finding container e8b90d47c151794823be869769ce846598c98c430d39f395f24a7f4da4856989: Status 404 returned error can't find the container with id e8b90d47c151794823be869769ce846598c98c430d39f395f24a7f4da4856989
Nov 02 01:44:13.576013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.575989    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"]
Nov 02 01:44:13.809809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.809772    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq" event={"ID":"49ae4474-4f10-47b8-86a3-d87b2f6e6021","Type":"ContainerStarted","Data":"c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef"}
Nov 02 01:44:13.810185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.809842    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq" event={"ID":"49ae4474-4f10-47b8-86a3-d87b2f6e6021","Type":"ContainerStarted","Data":"e8b90d47c151794823be869769ce846598c98c430d39f395f24a7f4da4856989"}
Nov 02 01:44:13.810508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.810433    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:44:13.812451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.812424    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm" event={"ID":"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655","Type":"ContainerStarted","Data":"724709d1d37caf0fa31958b9446a70941e22a7cfd83cfa6549509b73195d2c10"}
Nov 02 01:44:13.812600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.812489    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:44:13.814423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.814400    4851 generic.go:356] "Generic (PLEG): container finished" podID="9b1f6375-0a3f-4053-8398-289a711be8ac" containerID="686e7f34449c4cfaee7367d23925e7bc99c59c8165fc718fefaef9dce7cf642e" exitCode=0
Nov 02 01:44:13.814540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.814455    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m" event={"ID":"9b1f6375-0a3f-4053-8398-289a711be8ac","Type":"ContainerDied","Data":"686e7f34449c4cfaee7367d23925e7bc99c59c8165fc718fefaef9dce7cf642e"}
Nov 02 01:44:13.816269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.816243    4851 generic.go:356] "Generic (PLEG): container finished" podID="4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" containerID="0be195cf93747fd247b588c8c82c5ba4bbccdcefe86e3128c35589133c548653" exitCode=0
Nov 02 01:44:13.816405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.816320    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns" event={"ID":"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8","Type":"ContainerDied","Data":"0be195cf93747fd247b588c8c82c5ba4bbccdcefe86e3128c35589133c548653"}
Nov 02 01:44:13.833905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.833822    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq" podStartSLOduration=1.833803495 podStartE2EDuration="1.833803495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:13.830160939 +0000 UTC m=+1759.135093766" watchObservedRunningTime="2026-11-02 01:44:13.833803495 +0000 UTC m=+1759.138736323"
Nov 02 01:44:13.894895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.894831    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm" podStartSLOduration=1.894811274 podStartE2EDuration="1.894811274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:13.891940676 +0000 UTC m=+1759.196873500" watchObservedRunningTime="2026-11-02 01:44:13.894811274 +0000 UTC m=+1759.199744099"
Nov 02 01:44:13.909418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.909389    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:44:13.953064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.953034    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-kube-api-access-vrtnz\" (UniqueName: \"kubernetes.io/projected/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-kube-api-access-vrtnz\") pod \"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8\" (UID: \"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8\") "
Nov 02 01:44:13.953244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.953080    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-plugins-config-volume\") pod \"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8\" (UID: \"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8\") "
Nov 02 01:44:13.953607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.953569    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-plugins-config-volume" pod "4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" (UID: "4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:13.954251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.954228    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:13.955816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:13.955792    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-kube-api-access-vrtnz" pod "4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" (UID: "4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8"). InnerVolumeSpecName "kube-api-access-vrtnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:14.060372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:14.060178    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrtnz\" (UniqueName: \"kubernetes.io/projected/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8-kube-api-access-vrtnz\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:14.826180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:14.826151    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm" event={"ID":"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655","Type":"ContainerStarted","Data":"a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c"}
Nov 02 01:44:14.827768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:14.827743    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"
Nov 02 01:44:14.827901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:14.827790    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns" event={"ID":"4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8","Type":"ContainerDied","Data":"09681f92c31952c3e575197b4da8c33896352a4bc9382989289c1e4f6f649cae"}
Nov 02 01:44:14.827901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:14.827845    4851 scope.go:122] "RemoveContainer" containerID="0be195cf93747fd247b588c8c82c5ba4bbccdcefe86e3128c35589133c548653"
Nov 02 01:44:14.856985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:14.856954    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"]
Nov 02 01:44:14.864123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:14.864093    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffe5c1-847849fbc-g7pns"]
Nov 02 01:44:15.024727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.024700    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:15.187156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.187126    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-openshift-ca\") pod \"9b1f6375-0a3f-4053-8398-289a711be8ac\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") "
Nov 02 01:44:15.187353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.187250    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-xks-ca\") pod \"9b1f6375-0a3f-4053-8398-289a711be8ac\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") "
Nov 02 01:44:15.187353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.187319    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1f6375-0a3f-4053-8398-289a711be8ac-kube-api-access-gznqd\" (UniqueName: \"kubernetes.io/projected/9b1f6375-0a3f-4053-8398-289a711be8ac-kube-api-access-gznqd\") pod \"9b1f6375-0a3f-4053-8398-289a711be8ac\" (UID: \"9b1f6375-0a3f-4053-8398-289a711be8ac\") "
Nov 02 01:44:15.187486 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.187465    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-openshift-ca" pod "9b1f6375-0a3f-4053-8398-289a711be8ac" (UID: "9b1f6375-0a3f-4053-8398-289a711be8ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:15.187840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.187821    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:15.189535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.189510    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1f6375-0a3f-4053-8398-289a711be8ac-kube-api-access-gznqd" pod "9b1f6375-0a3f-4053-8398-289a711be8ac" (UID: "9b1f6375-0a3f-4053-8398-289a711be8ac"). InnerVolumeSpecName "kube-api-access-gznqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:15.189620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.189546    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-xks-ca" pod "9b1f6375-0a3f-4053-8398-289a711be8ac" (UID: "9b1f6375-0a3f-4053-8398-289a711be8ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:44:15.289509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.289478    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1f6375-0a3f-4053-8398-289a711be8ac-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:15.289509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.289508    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gznqd\" (UniqueName: \"kubernetes.io/projected/9b1f6375-0a3f-4053-8398-289a711be8ac-kube-api-access-gznqd\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:15.838763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.838729    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"
Nov 02 01:44:15.838763 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.838742    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m" event={"ID":"9b1f6375-0a3f-4053-8398-289a711be8ac","Type":"ContainerDied","Data":"f6ed21188ef80f7665946dae20f76bbee0df743ed2d1bda3ed9b34f6490d33e4"}
Nov 02 01:44:15.839326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.838789    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6ed21188ef80f7665946dae20f76bbee0df743ed2d1bda3ed9b34f6490d33e4"
Nov 02 01:44:15.883801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.883762    4851 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:44:15.883943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:15.883819    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:44:16.306840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.306805    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"]
Nov 02 01:44:16.311423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.311392    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"]
Nov 02 01:44:16.850742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.850696    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" path="/var/lib/kubelet/pods/4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8/volumes"
Nov 02 01:44:16.852035 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.852016    4851 generic.go:356] "Generic (PLEG): container finished" podID="099e6161-f6ee-4eb9-b200-715180f65227" containerID="2b681c18b956eaa5c0a9db4063765ff4564890e1b9e1ae9726686a31d891e5df" exitCode=0
Nov 02 01:44:16.852130 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.852091    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" event={"ID":"099e6161-f6ee-4eb9-b200-715180f65227","Type":"ContainerDied","Data":"2b681c18b956eaa5c0a9db4063765ff4564890e1b9e1ae9726686a31d891e5df"}
Nov 02 01:44:16.852173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.852139    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s" event={"ID":"099e6161-f6ee-4eb9-b200-715180f65227","Type":"ContainerDied","Data":"5b2d079aaec005105e6b84bf00da7ffcaec9751a910852a784b363d154117962"}
Nov 02 01:44:16.852173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.852157    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b2d079aaec005105e6b84bf00da7ffcaec9751a910852a784b363d154117962"
Nov 02 01:44:16.868109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.868089    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:44:16.878276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.878249    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"]
Nov 02 01:44:16.883579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.883554    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-47dd19-d82v8"]
Nov 02 01:44:16.936336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.936281    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-credential-socket\") pod \"099e6161-f6ee-4eb9-b200-715180f65227\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") "
Nov 02 01:44:16.936539 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.936386    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/099e6161-f6ee-4eb9-b200-715180f65227-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099e6161-f6ee-4eb9-b200-715180f65227-istio-podinfo\") pod \"099e6161-f6ee-4eb9-b200-715180f65227\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") "
Nov 02 01:44:16.936539 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.936461    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099e6161-f6ee-4eb9-b200-715180f65227-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099e6161-f6ee-4eb9-b200-715180f65227-istiod-ca-cert\") pod \"099e6161-f6ee-4eb9-b200-715180f65227\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") "
Nov 02 01:44:16.936539 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.936516    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-certs\") pod \"099e6161-f6ee-4eb9-b200-715180f65227\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") "
Nov 02 01:44:16.936714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.936582    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-envoy\") pod \"099e6161-f6ee-4eb9-b200-715180f65227\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") "
Nov 02 01:44:16.936714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.936594    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-credential-socket" pod "099e6161-f6ee-4eb9-b200-715180f65227" (UID: "099e6161-f6ee-4eb9-b200-715180f65227"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:16.936714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.936669    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-socket\") pod \"099e6161-f6ee-4eb9-b200-715180f65227\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") "
Nov 02 01:44:16.936714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.936704    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-kube-api-access-vkbtl\" (UniqueName: \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-kube-api-access-vkbtl\") pod \"099e6161-f6ee-4eb9-b200-715180f65227\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") "
Nov 02 01:44:16.936921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.936906    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-data\") pod \"099e6161-f6ee-4eb9-b200-715180f65227\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") "
Nov 02 01:44:16.936978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.936956    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-istio-token\" (UniqueName: \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-istio-token\") pod \"099e6161-f6ee-4eb9-b200-715180f65227\" (UID: \"099e6161-f6ee-4eb9-b200-715180f65227\") "
Nov 02 01:44:16.937077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.937036    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099e6161-f6ee-4eb9-b200-715180f65227-istiod-ca-cert" pod "099e6161-f6ee-4eb9-b200-715180f65227" (UID: "099e6161-f6ee-4eb9-b200-715180f65227"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:16.937566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.937374    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-data" pod "099e6161-f6ee-4eb9-b200-715180f65227" (UID: "099e6161-f6ee-4eb9-b200-715180f65227"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:16.937566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.937527    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-socket" pod "099e6161-f6ee-4eb9-b200-715180f65227" (UID: "099e6161-f6ee-4eb9-b200-715180f65227"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:16.938477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.938099    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:16.938477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.938122    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:16.938477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.938136    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099e6161-f6ee-4eb9-b200-715180f65227-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:16.938477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.938149    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:16.938878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.938853    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-certs" pod "099e6161-f6ee-4eb9-b200-715180f65227" (UID: "099e6161-f6ee-4eb9-b200-715180f65227"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:16.939959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.939927    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-kube-api-access-vkbtl" pod "099e6161-f6ee-4eb9-b200-715180f65227" (UID: "099e6161-f6ee-4eb9-b200-715180f65227"). InnerVolumeSpecName "kube-api-access-vkbtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:16.940907 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.940881    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-envoy" pod "099e6161-f6ee-4eb9-b200-715180f65227" (UID: "099e6161-f6ee-4eb9-b200-715180f65227"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:16.941506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.941483    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-istio-token" pod "099e6161-f6ee-4eb9-b200-715180f65227" (UID: "099e6161-f6ee-4eb9-b200-715180f65227"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:16.941671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:16.941649    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/099e6161-f6ee-4eb9-b200-715180f65227-istio-podinfo" pod "099e6161-f6ee-4eb9-b200-715180f65227" (UID: "099e6161-f6ee-4eb9-b200-715180f65227"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:44:17.054087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:17.054049    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:17.054333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:17.054313    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbtl\" (UniqueName: \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-kube-api-access-vkbtl\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:17.058594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:17.058558    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099e6161-f6ee-4eb9-b200-715180f65227-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:17.058858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:17.058805    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099e6161-f6ee-4eb9-b200-715180f65227-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:17.058858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:17.058829    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099e6161-f6ee-4eb9-b200-715180f65227-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:17.598314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:17.598273    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:44:17.670606 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:17.670566    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:44:17.889591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:17.881460    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"
Nov 02 01:44:17.934332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:17.933848    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"]
Nov 02 01:44:17.946799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:17.946765    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-zcr2s"]
Nov 02 01:44:18.849330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:18.849263    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099e6161-f6ee-4eb9-b200-715180f65227" path="/var/lib/kubelet/pods/099e6161-f6ee-4eb9-b200-715180f65227/volumes"
Nov 02 01:44:18.849884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:18.849867    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c36349-4646-4bcd-9b40-ede290c97edc" path="/var/lib/kubelet/pods/d9c36349-4646-4bcd-9b40-ede290c97edc/volumes"
Nov 02 01:44:19.674366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:19.674334    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:44:20.352478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.352446    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"]
Nov 02 01:44:20.353256 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.353236    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9b1f6375-0a3f-4053-8398-289a711be8ac" containerName="revoke-keys"
Nov 02 01:44:20.353256 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.353256    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1f6375-0a3f-4053-8398-289a711be8ac" containerName="revoke-keys"
Nov 02 01:44:20.353454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.353336    4851 cpu_manager.go:414] "containerMap: removing container" podUID="4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" containerName="payload-pre-processing"
Nov 02 01:44:20.353454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.353345    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" containerName="payload-pre-processing"
Nov 02 01:44:20.353454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.353356    4851 cpu_manager.go:414] "containerMap: removing container" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy"
Nov 02 01:44:20.353454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.353363    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy"
Nov 02 01:44:20.353597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.353488    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1f6375-0a3f-4053-8398-289a711be8ac" containerName="revoke-keys"
Nov 02 01:44:20.353597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.353509    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="4084a05c-f8e7-4e06-9aa0-1e7e8f400dd8" containerName="payload-pre-processing"
Nov 02 01:44:20.353597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.353517    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="099e6161-f6ee-4eb9-b200-715180f65227" containerName="istio-proxy"
Nov 02 01:44:20.358658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.358631    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.372887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.372848    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:44:20.373732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.373704    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"]
Nov 02 01:44:20.374078 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.374060    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 01:44:20.376056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.374646    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8qwrd\""
Nov 02 01:44:20.376056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.374712    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-eaec8c-serving-cert\""
Nov 02 01:44:20.420916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.420861    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.421113 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.420923    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzm2\" (UniqueName: \"kubernetes.io/projected/19c91b78-6c4c-4802-95e8-c9a80956681b-kube-api-access-xpzm2\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.421113 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.421091    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c91b78-6c4c-4802-95e8-c9a80956681b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.421253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.421173    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.466171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.466135    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:44:20.549535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.541373    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.549535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.541675    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.549535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.541717    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzm2\" (UniqueName: \"kubernetes.io/projected/19c91b78-6c4c-4802-95e8-c9a80956681b-kube-api-access-xpzm2\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.549535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.541920    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c91b78-6c4c-4802-95e8-c9a80956681b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.549535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.542982    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c91b78-6c4c-4802-95e8-c9a80956681b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.591640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.575571    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.591640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.588166    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzm2\" (UniqueName: \"kubernetes.io/projected/19c91b78-6c4c-4802-95e8-c9a80956681b-kube-api-access-xpzm2\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.591640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.588771    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:20.683747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:20.683714    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:21.039463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:21.038530    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"]
Nov 02 01:44:21.799313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:21.792375    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"]
Nov 02 01:44:22.003044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:22.003005    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564" event={"ID":"19c91b78-6c4c-4802-95e8-c9a80956681b","Type":"ContainerStarted","Data":"1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57"}
Nov 02 01:44:22.003265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:22.003077    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:22.003265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:22.003100    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564" event={"ID":"19c91b78-6c4c-4802-95e8-c9a80956681b","Type":"ContainerStarted","Data":"750d655318a4ad94de7d6d6609abafb511923873b3250211f0627cc8ae64cc48"}
Nov 02 01:44:22.027813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:22.027756    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564" podStartSLOduration=1.483774729 podStartE2EDuration="2.027741449s" totalImagesPullingTime="543.96672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:22.023766067 +0000 UTC m=+1767.328698893" watchObservedRunningTime="2026-11-02 01:44:22.027741449 +0000 UTC m=+1767.332674274"
Nov 02 01:44:23.511320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.510972    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"]
Nov 02 01:44:23.525318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.522564    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:23.536313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.527523    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:44:23.536313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.528672    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:44:23.536313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.535259    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"]
Nov 02 01:44:23.617915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.617877    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d167dadf-5rlrg\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:23.617915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.617919    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5nq\" (UniqueName: \"kubernetes.io/projected/a9bd2122-f60f-47c4-b6ce-13a555092662-kube-api-access-zb5nq\") pod \"maas-api-revoke-sub-d167dadf-5rlrg\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:23.618172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.618157    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d167dadf-5rlrg\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:23.719692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.719658    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d167dadf-5rlrg\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:23.719872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.719700    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5nq\" (UniqueName: \"kubernetes.io/projected/a9bd2122-f60f-47c4-b6ce-13a555092662-kube-api-access-zb5nq\") pod \"maas-api-revoke-sub-d167dadf-5rlrg\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:23.719924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.719904    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d167dadf-5rlrg\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:23.720529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.720508    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d167dadf-5rlrg\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:23.722265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.722242    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d167dadf-5rlrg\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:23.732303 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.732261    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5nq\" (UniqueName: \"kubernetes.io/projected/a9bd2122-f60f-47c4-b6ce-13a555092662-kube-api-access-zb5nq\") pod \"maas-api-revoke-sub-d167dadf-5rlrg\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:23.876046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:23.875970    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:24.171855 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:24.171822    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"]
Nov 02 01:44:24.173550 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:24.173525    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bd2122_f60f_47c4_b6ce_13a555092662.slice/crio-536ac9fb996b5748a99d789e1876c743d54e3a6652dd8c247e0732fd5369710b WatchSource:0}: Error finding container 536ac9fb996b5748a99d789e1876c743d54e3a6652dd8c247e0732fd5369710b: Status 404 returned error can't find the container with id 536ac9fb996b5748a99d789e1876c743d54e3a6652dd8c247e0732fd5369710b
Nov 02 01:44:24.831513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:24.831437    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:44:25.033550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:25.033505    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg" event={"ID":"a9bd2122-f60f-47c4-b6ce-13a555092662","Type":"ContainerStarted","Data":"d0a6286ce85e8e1bcfc6d16bb8a80661299f8f5bb1f55f7b3fe1d0ee40402d44"}
Nov 02 01:44:25.033978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:25.033579    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg" event={"ID":"a9bd2122-f60f-47c4-b6ce-13a555092662","Type":"ContainerStarted","Data":"536ac9fb996b5748a99d789e1876c743d54e3a6652dd8c247e0732fd5369710b"}
Nov 02 01:44:25.057888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:25.057825    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg" podStartSLOduration=2.057802152 podStartE2EDuration="2.057802152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:25.052442168 +0000 UTC m=+1770.357374996" watchObservedRunningTime="2026-11-02 01:44:25.057802152 +0000 UTC m=+1770.362734979"
Nov 02 01:44:25.527267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:25.527220    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"]
Nov 02 01:44:25.527810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:25.527780    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy" containerID="cri-o://fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68" gracePeriod=30
Nov 02 01:44:25.872464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:25.857371    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:44:26.043673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.043641    4851 generic.go:356] "Generic (PLEG): container finished" podID="a9bd2122-f60f-47c4-b6ce-13a555092662" containerID="d0a6286ce85e8e1bcfc6d16bb8a80661299f8f5bb1f55f7b3fe1d0ee40402d44" exitCode=0
Nov 02 01:44:26.043850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.043724    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg" event={"ID":"a9bd2122-f60f-47c4-b6ce-13a555092662","Type":"ContainerDied","Data":"d0a6286ce85e8e1bcfc6d16bb8a80661299f8f5bb1f55f7b3fe1d0ee40402d44"}
Nov 02 01:44:26.792787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.792745    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc" podUID="41550ac8-635a-4162-bbce-4145804093be" containerName="payload-pre-processing" containerID="cri-o://715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c" gracePeriod=25
Nov 02 01:44:26.837443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.837407    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"]
Nov 02 01:44:26.843315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.843244    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:26.855799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.853728    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-3c4ea5-serving-cert\""
Nov 02 01:44:26.866267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.866226    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"]
Nov 02 01:44:26.904272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.904245    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:26.904704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.904362    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62wz8\" (UniqueName: \"kubernetes.io/projected/96064425-7a87-45ac-83a7-36fba9bb924d-kube-api-access-62wz8\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:26.904704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.904457    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:26.904704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:26.904525    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96064425-7a87-45ac-83a7-36fba9bb924d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:27.005945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.005908    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62wz8\" (UniqueName: \"kubernetes.io/projected/96064425-7a87-45ac-83a7-36fba9bb924d-kube-api-access-62wz8\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:27.005945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.005949    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:27.006167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.005983    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96064425-7a87-45ac-83a7-36fba9bb924d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:27.006167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.006150    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:27.006815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.006788    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96064425-7a87-45ac-83a7-36fba9bb924d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:27.008670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.008649    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:27.008796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.008705    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:27.018754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.018729    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62wz8\" (UniqueName: \"kubernetes.io/projected/96064425-7a87-45ac-83a7-36fba9bb924d-kube-api-access-62wz8\") pod \"maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:27.287437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.285260    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:27.366559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.366525    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:27.580375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.542313    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bd2122-f60f-47c4-b6ce-13a555092662-kube-api-access-zb5nq\" (UniqueName: \"kubernetes.io/projected/a9bd2122-f60f-47c4-b6ce-13a555092662-kube-api-access-zb5nq\") pod \"a9bd2122-f60f-47c4-b6ce-13a555092662\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") "
Nov 02 01:44:27.580755 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.580715    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-xks-ca\") pod \"a9bd2122-f60f-47c4-b6ce-13a555092662\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") "
Nov 02 01:44:27.581128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.581104    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-openshift-ca\") pod \"a9bd2122-f60f-47c4-b6ce-13a555092662\" (UID: \"a9bd2122-f60f-47c4-b6ce-13a555092662\") "
Nov 02 01:44:27.607343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.598058    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-xks-ca" pod "a9bd2122-f60f-47c4-b6ce-13a555092662" (UID: "a9bd2122-f60f-47c4-b6ce-13a555092662"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:44:27.607933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.607902    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-openshift-ca" pod "a9bd2122-f60f-47c4-b6ce-13a555092662" (UID: "a9bd2122-f60f-47c4-b6ce-13a555092662"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:27.618890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.618692    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bd2122-f60f-47c4-b6ce-13a555092662-kube-api-access-zb5nq" pod "a9bd2122-f60f-47c4-b6ce-13a555092662" (UID: "a9bd2122-f60f-47c4-b6ce-13a555092662"). InnerVolumeSpecName "kube-api-access-zb5nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:27.650369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.650333    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"]
Nov 02 01:44:27.651029 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:27.651005    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96064425_7a87_45ac_83a7_36fba9bb924d.slice/crio-8c64064bc5b2887c88a5e65e4216e2921f8dc97dc4a4dd0d8093a2f7dfa6cb25 WatchSource:0}: Error finding container 8c64064bc5b2887c88a5e65e4216e2921f8dc97dc4a4dd0d8093a2f7dfa6cb25: Status 404 returned error can't find the container with id 8c64064bc5b2887c88a5e65e4216e2921f8dc97dc4a4dd0d8093a2f7dfa6cb25
Nov 02 01:44:27.695429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.695394    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb5nq\" (UniqueName: \"kubernetes.io/projected/a9bd2122-f60f-47c4-b6ce-13a555092662-kube-api-access-zb5nq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:27.697211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.697170    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:27.697687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:27.697668    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9bd2122-f60f-47c4-b6ce-13a555092662-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:28.018888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:28.018858    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:44:28.066837 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:28.066807    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv" event={"ID":"96064425-7a87-45ac-83a7-36fba9bb924d","Type":"ContainerStarted","Data":"8c64064bc5b2887c88a5e65e4216e2921f8dc97dc4a4dd0d8093a2f7dfa6cb25"}
Nov 02 01:44:28.068615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:28.068587    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg" event={"ID":"a9bd2122-f60f-47c4-b6ce-13a555092662","Type":"ContainerDied","Data":"536ac9fb996b5748a99d789e1876c743d54e3a6652dd8c247e0732fd5369710b"}
Nov 02 01:44:28.068751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:28.068638    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="536ac9fb996b5748a99d789e1876c743d54e3a6652dd8c247e0732fd5369710b"
Nov 02 01:44:28.068751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:28.068644    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"
Nov 02 01:44:29.088831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:29.088795    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv" event={"ID":"96064425-7a87-45ac-83a7-36fba9bb924d","Type":"ContainerStarted","Data":"fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6"}
Nov 02 01:44:29.089282 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:29.088919    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:29.114316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:29.114255    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv" podStartSLOduration=2.704056997 podStartE2EDuration="3.114241894s" totalImagesPullingTime="410.184897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:29.109955045 +0000 UTC m=+1774.414887881" watchObservedRunningTime="2026-11-02 01:44:29.114241894 +0000 UTC m=+1774.419174720"
Nov 02 01:44:30.836450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.836422    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:44:30.975962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.975923    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-data\") pod \"ffcb2a59-eee1-4301-89b7-828d25dd269d\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") "
Nov 02 01:44:30.976142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.976000    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-kube-api-access-pdxjv\" (UniqueName: \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-kube-api-access-pdxjv\") pod \"ffcb2a59-eee1-4301-89b7-828d25dd269d\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") "
Nov 02 01:44:30.976142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.976077    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-certs\") pod \"ffcb2a59-eee1-4301-89b7-828d25dd269d\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") "
Nov 02 01:44:30.976142 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.976110    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcb2a59-eee1-4301-89b7-828d25dd269d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcb2a59-eee1-4301-89b7-828d25dd269d-istiod-ca-cert\") pod \"ffcb2a59-eee1-4301-89b7-828d25dd269d\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") "
Nov 02 01:44:30.976356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.976181    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-socket\") pod \"ffcb2a59-eee1-4301-89b7-828d25dd269d\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") "
Nov 02 01:44:30.976356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.976279    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-podinfo\") pod \"ffcb2a59-eee1-4301-89b7-828d25dd269d\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") "
Nov 02 01:44:30.976521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.976476    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-credential-socket\") pod \"ffcb2a59-eee1-4301-89b7-828d25dd269d\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") "
Nov 02 01:44:30.976961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.976899    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-data" pod "ffcb2a59-eee1-4301-89b7-828d25dd269d" (UID: "ffcb2a59-eee1-4301-89b7-828d25dd269d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:30.977108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.977031    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-envoy\") pod \"ffcb2a59-eee1-4301-89b7-828d25dd269d\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") "
Nov 02 01:44:30.977255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.977233    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-token\" (UniqueName: \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-token\") pod \"ffcb2a59-eee1-4301-89b7-828d25dd269d\" (UID: \"ffcb2a59-eee1-4301-89b7-828d25dd269d\") "
Nov 02 01:44:30.977874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.977844    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-socket" pod "ffcb2a59-eee1-4301-89b7-828d25dd269d" (UID: "ffcb2a59-eee1-4301-89b7-828d25dd269d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:30.978355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.978331    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:30.978483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.978358    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:30.978759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.978731    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcb2a59-eee1-4301-89b7-828d25dd269d-istiod-ca-cert" pod "ffcb2a59-eee1-4301-89b7-828d25dd269d" (UID: "ffcb2a59-eee1-4301-89b7-828d25dd269d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:30.979201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.979009    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-certs" pod "ffcb2a59-eee1-4301-89b7-828d25dd269d" (UID: "ffcb2a59-eee1-4301-89b7-828d25dd269d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:30.979201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.979167    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-credential-socket" pod "ffcb2a59-eee1-4301-89b7-828d25dd269d" (UID: "ffcb2a59-eee1-4301-89b7-828d25dd269d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:30.980426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.980400    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-envoy" pod "ffcb2a59-eee1-4301-89b7-828d25dd269d" (UID: "ffcb2a59-eee1-4301-89b7-828d25dd269d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:30.980548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.980441    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-kube-api-access-pdxjv" pod "ffcb2a59-eee1-4301-89b7-828d25dd269d" (UID: "ffcb2a59-eee1-4301-89b7-828d25dd269d"). InnerVolumeSpecName "kube-api-access-pdxjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:30.985801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.985772    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-podinfo" pod "ffcb2a59-eee1-4301-89b7-828d25dd269d" (UID: "ffcb2a59-eee1-4301-89b7-828d25dd269d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:44:30.986006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:30.985987    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-token" pod "ffcb2a59-eee1-4301-89b7-828d25dd269d" (UID: "ffcb2a59-eee1-4301-89b7-828d25dd269d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:31.080936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.080744    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.080936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.080785    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.080936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.080801    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.080936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.080816    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdxjv\" (UniqueName: \"kubernetes.io/projected/ffcb2a59-eee1-4301-89b7-828d25dd269d-kube-api-access-pdxjv\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.080936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.080831    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffcb2a59-eee1-4301-89b7-828d25dd269d-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.080936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.080844    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffcb2a59-eee1-4301-89b7-828d25dd269d-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.080936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.080857    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffcb2a59-eee1-4301-89b7-828d25dd269d-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.232344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.231008    4851 generic.go:356] "Generic (PLEG): container finished" podID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerID="fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68" exitCode=0
Nov 02 01:44:31.232344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.231490    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" event={"ID":"ffcb2a59-eee1-4301-89b7-828d25dd269d","Type":"ContainerDied","Data":"fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68"}
Nov 02 01:44:31.232344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.231575    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s" event={"ID":"ffcb2a59-eee1-4301-89b7-828d25dd269d","Type":"ContainerDied","Data":"b021349c17514e6b351a71463507e8fefa5d647a262404063eacbe97492b73f5"}
Nov 02 01:44:31.232344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.231623    4851 scope.go:122] "RemoveContainer" containerID="fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68"
Nov 02 01:44:31.232344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.231871    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"
Nov 02 01:44:31.342019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.337844    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" containerID="cri-o://46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4" gracePeriod=45
Nov 02 01:44:31.345939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.345877    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" containerID="cri-o://5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2" gracePeriod=45
Nov 02 01:44:31.371422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.371388    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"]
Nov 02 01:44:31.371581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.371451    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-vsv4s"]
Nov 02 01:44:31.377828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.374515    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": EOF"
Nov 02 01:44:31.396466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.394824    4851 scope.go:122] "RemoveContainer" containerID="fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68"
Nov 02 01:44:31.397519 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:31.397478    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68\": container with ID starting with fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68 not found: ID does not exist" containerID="fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68"
Nov 02 01:44:31.397671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.397533    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68"} err="failed to get container status \"fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68\": rpc error: code = NotFound desc = could not find container \"fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68\": container with ID starting with fd453738fa2bee5d4a3e02ca06ac54b37b518937b0d7c70362b8e00dbab5bd68 not found: ID does not exist"
Nov 02 01:44:31.669531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.668910    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:44:31.716037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.709142    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:44:31.716037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.714534    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc" podUID="41550ac8-635a-4162-bbce-4145804093be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:44:31.818595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818565    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94655091-3dc8-4b39-bcf2-a8fe7073b350-tls-certs\" (UniqueName: \"kubernetes.io/secret/94655091-3dc8-4b39-bcf2-a8fe7073b350-tls-certs\") pod \"94655091-3dc8-4b39-bcf2-a8fe7073b350\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") "
Nov 02 01:44:31.818800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818654    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-model-cache\") pod \"94655091-3dc8-4b39-bcf2-a8fe7073b350\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") "
Nov 02 01:44:31.818800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818687    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94655091-3dc8-4b39-bcf2-a8fe7073b350-kube-api-access-q8k2s\" (UniqueName: \"kubernetes.io/projected/94655091-3dc8-4b39-bcf2-a8fe7073b350-kube-api-access-q8k2s\") pod \"94655091-3dc8-4b39-bcf2-a8fe7073b350\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") "
Nov 02 01:44:31.818800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818743    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1afe6338-3fb4-4829-88f9-232fc8c32127-tls-certs\" (UniqueName: \"kubernetes.io/secret/1afe6338-3fb4-4829-88f9-232fc8c32127-tls-certs\") pod \"1afe6338-3fb4-4829-88f9-232fc8c32127\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") "
Nov 02 01:44:31.818800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818765    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-home\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-home\") pod \"1afe6338-3fb4-4829-88f9-232fc8c32127\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") "
Nov 02 01:44:31.818800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818791    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-dshm\") pod \"94655091-3dc8-4b39-bcf2-a8fe7073b350\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") "
Nov 02 01:44:31.819095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818834    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-dshm\") pod \"1afe6338-3fb4-4829-88f9-232fc8c32127\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") "
Nov 02 01:44:31.819095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818890    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-tmp-dir\") pod \"94655091-3dc8-4b39-bcf2-a8fe7073b350\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") "
Nov 02 01:44:31.819095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818931    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-tmp-dir\") pod \"1afe6338-3fb4-4829-88f9-232fc8c32127\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") "
Nov 02 01:44:31.819095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818975    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-home\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-home\") pod \"94655091-3dc8-4b39-bcf2-a8fe7073b350\" (UID: \"94655091-3dc8-4b39-bcf2-a8fe7073b350\") "
Nov 02 01:44:31.819095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.818986    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-model-cache" pod "94655091-3dc8-4b39-bcf2-a8fe7073b350" (UID: "94655091-3dc8-4b39-bcf2-a8fe7073b350"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:31.819095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.819023    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-model-cache\") pod \"1afe6338-3fb4-4829-88f9-232fc8c32127\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") "
Nov 02 01:44:31.819502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.819098    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-home" pod "1afe6338-3fb4-4829-88f9-232fc8c32127" (UID: "1afe6338-3fb4-4829-88f9-232fc8c32127"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:31.819502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.819116    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1afe6338-3fb4-4829-88f9-232fc8c32127-kube-api-access-gqcps\" (UniqueName: \"kubernetes.io/projected/1afe6338-3fb4-4829-88f9-232fc8c32127-kube-api-access-gqcps\") pod \"1afe6338-3fb4-4829-88f9-232fc8c32127\" (UID: \"1afe6338-3fb4-4829-88f9-232fc8c32127\") "
Nov 02 01:44:31.819502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.819228    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-tmp-dir" pod "94655091-3dc8-4b39-bcf2-a8fe7073b350" (UID: "94655091-3dc8-4b39-bcf2-a8fe7073b350"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:31.819502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.819245    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-tmp-dir" pod "1afe6338-3fb4-4829-88f9-232fc8c32127" (UID: "1afe6338-3fb4-4829-88f9-232fc8c32127"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:31.819502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.819394    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-home" pod "94655091-3dc8-4b39-bcf2-a8fe7073b350" (UID: "94655091-3dc8-4b39-bcf2-a8fe7073b350"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:31.819793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.819624    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-model-cache" pod "1afe6338-3fb4-4829-88f9-232fc8c32127" (UID: "1afe6338-3fb4-4829-88f9-232fc8c32127"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:31.820110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.820085    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.820196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.820118    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.820196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.820134    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.820196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.820147    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.820196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.820160    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.820196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.820175    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.821624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.821598    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-dshm" pod "1afe6338-3fb4-4829-88f9-232fc8c32127" (UID: "1afe6338-3fb4-4829-88f9-232fc8c32127"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:31.821624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.821615    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1afe6338-3fb4-4829-88f9-232fc8c32127-tls-certs" pod "1afe6338-3fb4-4829-88f9-232fc8c32127" (UID: "1afe6338-3fb4-4829-88f9-232fc8c32127"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:44:31.821804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.821684    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94655091-3dc8-4b39-bcf2-a8fe7073b350-tls-certs" pod "94655091-3dc8-4b39-bcf2-a8fe7073b350" (UID: "94655091-3dc8-4b39-bcf2-a8fe7073b350"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:44:31.822018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.821991    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-dshm" pod "94655091-3dc8-4b39-bcf2-a8fe7073b350" (UID: "94655091-3dc8-4b39-bcf2-a8fe7073b350"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:31.824704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.824676    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afe6338-3fb4-4829-88f9-232fc8c32127-kube-api-access-gqcps" pod "1afe6338-3fb4-4829-88f9-232fc8c32127" (UID: "1afe6338-3fb4-4829-88f9-232fc8c32127"). InnerVolumeSpecName "kube-api-access-gqcps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:31.824835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.824821    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94655091-3dc8-4b39-bcf2-a8fe7073b350-kube-api-access-q8k2s" pod "94655091-3dc8-4b39-bcf2-a8fe7073b350" (UID: "94655091-3dc8-4b39-bcf2-a8fe7073b350"). InnerVolumeSpecName "kube-api-access-q8k2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:31.922005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.921973    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqcps\" (UniqueName: \"kubernetes.io/projected/1afe6338-3fb4-4829-88f9-232fc8c32127-kube-api-access-gqcps\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.922005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.922002    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94655091-3dc8-4b39-bcf2-a8fe7073b350-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.922415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.922014    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8k2s\" (UniqueName: \"kubernetes.io/projected/94655091-3dc8-4b39-bcf2-a8fe7073b350-kube-api-access-q8k2s\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.922415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.922024    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1afe6338-3fb4-4829-88f9-232fc8c32127-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.922415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.922034    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94655091-3dc8-4b39-bcf2-a8fe7073b350-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:31.922415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:31.922042    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1afe6338-3fb4-4829-88f9-232fc8c32127-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:32.310994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.310841    4851 generic.go:356] "Generic (PLEG): container finished" podID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerID="5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2" exitCode=0
Nov 02 01:44:32.310994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.310886    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" event={"ID":"94655091-3dc8-4b39-bcf2-a8fe7073b350","Type":"ContainerDied","Data":"5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2"}
Nov 02 01:44:32.310994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.310925    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"
Nov 02 01:44:32.310994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.310957    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" event={"ID":"94655091-3dc8-4b39-bcf2-a8fe7073b350","Type":"ContainerDied","Data":"ebb781aa640fd1ba77a8c3efa56339da67fe4106c5311543edf59035f4b9fdc3"}
Nov 02 01:44:32.311383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.310982    4851 scope.go:122] "RemoveContainer" containerID="5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2"
Nov 02 01:44:32.313898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.313872    4851 generic.go:356] "Generic (PLEG): container finished" podID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerID="46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4" exitCode=0
Nov 02 01:44:32.314051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.313984    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" event={"ID":"1afe6338-3fb4-4829-88f9-232fc8c32127","Type":"ContainerDied","Data":"46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4"}
Nov 02 01:44:32.314051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.314010    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf" event={"ID":"1afe6338-3fb4-4829-88f9-232fc8c32127","Type":"ContainerDied","Data":"0049eb4585bb3222e4e0c7b5344e791c49e9735ad53943be62087bc6d28eae07"}
Nov 02 01:44:32.314227 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.314058    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"
Nov 02 01:44:32.326831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.326815    4851 scope.go:122] "RemoveContainer" containerID="5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2"
Nov 02 01:44:32.327191 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:32.327167    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2\": container with ID starting with 5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2 not found: ID does not exist" containerID="5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2"
Nov 02 01:44:32.327262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.327204    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2"} err="failed to get container status \"5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2\": rpc error: code = NotFound desc = could not find container \"5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2\": container with ID starting with 5a88d1a995cd23b69fe73b18d8e8631c38b30eeefe6a74ba6a67ebce0c123aa2 not found: ID does not exist"
Nov 02 01:44:32.327262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.327228    4851 scope.go:122] "RemoveContainer" containerID="46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4"
Nov 02 01:44:32.340101 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.340041    4851 scope.go:122] "RemoveContainer" containerID="46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4"
Nov 02 01:44:32.341936 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:32.340829    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4\": container with ID starting with 46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4 not found: ID does not exist" containerID="46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4"
Nov 02 01:44:32.341936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.340864    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4"} err="failed to get container status \"46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4\": rpc error: code = NotFound desc = could not find container \"46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4\": container with ID starting with 46a0b5e546da9d988ae3d712a938f09bb4e7bd21f820efaf2d94f01e516690a4 not found: ID does not exist"
Nov 02 01:44:32.344422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.344402    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"]
Nov 02 01:44:32.350845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.350823    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm"]
Nov 02 01:44:32.358822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.358800    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"]
Nov 02 01:44:32.369020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.368998    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/premium-w4-b6ec07-kserve-5cbb9bf76f-flkgf"]
Nov 02 01:44:32.560951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.560806    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/simulator-w4-b6ec07-kserve-74cd5bff5b-rfgvm" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:44:32.850074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.850039    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" path="/var/lib/kubelet/pods/1afe6338-3fb4-4829-88f9-232fc8c32127/volumes"
Nov 02 01:44:32.850743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.850721    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" path="/var/lib/kubelet/pods/94655091-3dc8-4b39-bcf2-a8fe7073b350/volumes"
Nov 02 01:44:32.851233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:32.851214    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" path="/var/lib/kubelet/pods/ffcb2a59-eee1-4301-89b7-828d25dd269d/volumes"
Nov 02 01:44:34.805316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.804669    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"]
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.822974    4851 cpu_manager.go:414] "containerMap: removing container" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823007    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823054    4851 cpu_manager.go:414] "containerMap: removing container" podUID="a9bd2122-f60f-47c4-b6ce-13a555092662" containerName="revoke-keys"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823061    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bd2122-f60f-47c4-b6ce-13a555092662" containerName="revoke-keys"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823109    4851 cpu_manager.go:414] "containerMap: removing container" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823119    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823135    4851 cpu_manager.go:414] "containerMap: removing container" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823143    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823342    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="94655091-3dc8-4b39-bcf2-a8fe7073b350" containerName="main"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823371    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bd2122-f60f-47c4-b6ce-13a555092662" containerName="revoke-keys"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823391    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="1afe6338-3fb4-4829-88f9-232fc8c32127" containerName="main"
Nov 02 01:44:34.833316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.823400    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcb2a59-eee1-4301-89b7-828d25dd269d" containerName="istio-proxy"
Nov 02 01:44:34.882578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.873942    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:34.906313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.902021    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:44:34.927616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.917075    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:44:34.927616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.919070    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e74772-510a-4359-9d31-12f064d2258b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:34.927616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.919116    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e74772-510a-4359-9d31-12f064d2258b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:34.927616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.919152    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rv2s\" (UniqueName: \"kubernetes.io/projected/90e74772-510a-4359-9d31-12f064d2258b-kube-api-access-5rv2s\") pod \"maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:34.931912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:34.931874    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"]
Nov 02 01:44:35.020994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:35.020950    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e74772-510a-4359-9d31-12f064d2258b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:35.021220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:35.021039    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e74772-510a-4359-9d31-12f064d2258b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:35.021220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:35.021099    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rv2s\" (UniqueName: \"kubernetes.io/projected/90e74772-510a-4359-9d31-12f064d2258b-kube-api-access-5rv2s\") pod \"maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:35.021820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:35.021795    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e74772-510a-4359-9d31-12f064d2258b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:35.023885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:35.023863    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e74772-510a-4359-9d31-12f064d2258b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:35.037961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:35.037927    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rv2s\" (UniqueName: \"kubernetes.io/projected/90e74772-510a-4359-9d31-12f064d2258b-kube-api-access-5rv2s\") pod \"maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:35.104874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:35.104796    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:44:35.248262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:35.248223    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:35.540671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:35.540645    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"]
Nov 02 01:44:35.540892 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:35.540873    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e74772_510a_4359_9d31_12f064d2258b.slice/crio-cd908a8a6373c47e8d68950202a95d6776ba8ba7fc2bc7fd626ef68d314e44a0 WatchSource:0}: Error finding container cd908a8a6373c47e8d68950202a95d6776ba8ba7fc2bc7fd626ef68d314e44a0: Status 404 returned error can't find the container with id cd908a8a6373c47e8d68950202a95d6776ba8ba7fc2bc7fd626ef68d314e44a0
Nov 02 01:44:36.330754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.330719    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"]
Nov 02 01:44:36.335694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.335669    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:36.347975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.347941    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"]
Nov 02 01:44:36.373547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.373496    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg" event={"ID":"90e74772-510a-4359-9d31-12f064d2258b","Type":"ContainerStarted","Data":"3176538d307b519b10a82ae5be1b7610202baca8b4329f998c70e56a32af20bd"}
Nov 02 01:44:36.373955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.373914    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg" event={"ID":"90e74772-510a-4359-9d31-12f064d2258b","Type":"ContainerStarted","Data":"cd908a8a6373c47e8d68950202a95d6776ba8ba7fc2bc7fd626ef68d314e44a0"}
Nov 02 01:44:36.640966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.640864    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7f721a9-x7wl5\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:36.641404 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.641019    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcs4d\" (UniqueName: \"kubernetes.io/projected/eb611906-0a66-4656-8fa0-391c284c7c99-kube-api-access-kcs4d\") pod \"maas-api-revoke-sub-c7f721a9-x7wl5\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:36.641967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.641940    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7f721a9-x7wl5\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:36.743678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.743642    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7f721a9-x7wl5\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:36.743884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.743727    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7f721a9-x7wl5\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:36.743884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.743793    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcs4d\" (UniqueName: \"kubernetes.io/projected/eb611906-0a66-4656-8fa0-391c284c7c99-kube-api-access-kcs4d\") pod \"maas-api-revoke-sub-c7f721a9-x7wl5\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:36.744424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.744398    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7f721a9-x7wl5\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:36.746395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.746374    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7f721a9-x7wl5\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:36.756506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.756480    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcs4d\" (UniqueName: \"kubernetes.io/projected/eb611906-0a66-4656-8fa0-391c284c7c99-kube-api-access-kcs4d\") pod \"maas-api-revoke-sub-c7f721a9-x7wl5\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:36.972833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:36.971560    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:37.287824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:37.287766    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg" podStartSLOduration=3.2877507550000002 podStartE2EDuration="3.287750755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:36.624875839 +0000 UTC m=+1781.929808665" watchObservedRunningTime="2026-11-02 01:44:37.287750755 +0000 UTC m=+1782.592683616"
Nov 02 01:44:37.287824 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:37.287805    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb611906_0a66_4656_8fa0_391c284c7c99.slice/crio-1700e23e45de9c105801fc56437ee76ca3fdcc512c24d183d0498e8c1c3b3657 WatchSource:0}: Error finding container 1700e23e45de9c105801fc56437ee76ca3fdcc512c24d183d0498e8c1c3b3657: Status 404 returned error can't find the container with id 1700e23e45de9c105801fc56437ee76ca3fdcc512c24d183d0498e8c1c3b3657
Nov 02 01:44:37.288056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:37.288047    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"]
Nov 02 01:44:37.383960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:37.383922    4851 generic.go:356] "Generic (PLEG): container finished" podID="90e74772-510a-4359-9d31-12f064d2258b" containerID="3176538d307b519b10a82ae5be1b7610202baca8b4329f998c70e56a32af20bd" exitCode=0
Nov 02 01:44:37.384466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:37.384009    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg" event={"ID":"90e74772-510a-4359-9d31-12f064d2258b","Type":"ContainerDied","Data":"3176538d307b519b10a82ae5be1b7610202baca8b4329f998c70e56a32af20bd"}
Nov 02 01:44:37.385658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:37.385626    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5" event={"ID":"eb611906-0a66-4656-8fa0-391c284c7c99","Type":"ContainerStarted","Data":"1700e23e45de9c105801fc56437ee76ca3fdcc512c24d183d0498e8c1c3b3657"}
Nov 02 01:44:38.397249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.397216    4851 generic.go:356] "Generic (PLEG): container finished" podID="eb611906-0a66-4656-8fa0-391c284c7c99" containerID="b52e8c1ceee9c50e4d4a3ace5b0e125e428de75f5ee18c230a9c31aa1f894e1c" exitCode=0
Nov 02 01:44:38.397709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.397332    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5" event={"ID":"eb611906-0a66-4656-8fa0-391c284c7c99","Type":"ContainerDied","Data":"b52e8c1ceee9c50e4d4a3ace5b0e125e428de75f5ee18c230a9c31aa1f894e1c"}
Nov 02 01:44:38.664679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.664651    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:38.801489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.801450    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90e74772-510a-4359-9d31-12f064d2258b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e74772-510a-4359-9d31-12f064d2258b-maas-api-xks-ca\") pod \"90e74772-510a-4359-9d31-12f064d2258b\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") "
Nov 02 01:44:38.801667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.801654    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e74772-510a-4359-9d31-12f064d2258b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e74772-510a-4359-9d31-12f064d2258b-maas-api-openshift-ca\") pod \"90e74772-510a-4359-9d31-12f064d2258b\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") "
Nov 02 01:44:38.801749 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.801691    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e74772-510a-4359-9d31-12f064d2258b-kube-api-access-5rv2s\" (UniqueName: \"kubernetes.io/projected/90e74772-510a-4359-9d31-12f064d2258b-kube-api-access-5rv2s\") pod \"90e74772-510a-4359-9d31-12f064d2258b\" (UID: \"90e74772-510a-4359-9d31-12f064d2258b\") "
Nov 02 01:44:38.802052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.802018    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e74772-510a-4359-9d31-12f064d2258b-maas-api-openshift-ca" pod "90e74772-510a-4359-9d31-12f064d2258b" (UID: "90e74772-510a-4359-9d31-12f064d2258b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:38.802194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.802178    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e74772-510a-4359-9d31-12f064d2258b-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:38.803773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.803750    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e74772-510a-4359-9d31-12f064d2258b-maas-api-xks-ca" pod "90e74772-510a-4359-9d31-12f064d2258b" (UID: "90e74772-510a-4359-9d31-12f064d2258b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:44:38.804003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.803989    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e74772-510a-4359-9d31-12f064d2258b-kube-api-access-5rv2s" pod "90e74772-510a-4359-9d31-12f064d2258b" (UID: "90e74772-510a-4359-9d31-12f064d2258b"). InnerVolumeSpecName "kube-api-access-5rv2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:38.904096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.904006    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rv2s\" (UniqueName: \"kubernetes.io/projected/90e74772-510a-4359-9d31-12f064d2258b-kube-api-access-5rv2s\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:38.904096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:38.904044    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e74772-510a-4359-9d31-12f064d2258b-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:39.407263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.407222    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"
Nov 02 01:44:39.407723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.407263    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg" event={"ID":"90e74772-510a-4359-9d31-12f064d2258b","Type":"ContainerDied","Data":"cd908a8a6373c47e8d68950202a95d6776ba8ba7fc2bc7fd626ef68d314e44a0"}
Nov 02 01:44:39.407723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.407354    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd908a8a6373c47e8d68950202a95d6776ba8ba7fc2bc7fd626ef68d314e44a0"
Nov 02 01:44:39.588389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.588362    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:39.714778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.714745    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb611906-0a66-4656-8fa0-391c284c7c99-kube-api-access-kcs4d\" (UniqueName: \"kubernetes.io/projected/eb611906-0a66-4656-8fa0-391c284c7c99-kube-api-access-kcs4d\") pod \"eb611906-0a66-4656-8fa0-391c284c7c99\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") "
Nov 02 01:44:39.714986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.714805    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-openshift-ca\") pod \"eb611906-0a66-4656-8fa0-391c284c7c99\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") "
Nov 02 01:44:39.715125 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.715106    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-xks-ca\") pod \"eb611906-0a66-4656-8fa0-391c284c7c99\" (UID: \"eb611906-0a66-4656-8fa0-391c284c7c99\") "
Nov 02 01:44:39.715216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.715162    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-openshift-ca" pod "eb611906-0a66-4656-8fa0-391c284c7c99" (UID: "eb611906-0a66-4656-8fa0-391c284c7c99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:39.715723 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.715701    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:39.717133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.717099    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb611906-0a66-4656-8fa0-391c284c7c99-kube-api-access-kcs4d" pod "eb611906-0a66-4656-8fa0-391c284c7c99" (UID: "eb611906-0a66-4656-8fa0-391c284c7c99"). InnerVolumeSpecName "kube-api-access-kcs4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:39.717133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.717119    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-xks-ca" pod "eb611906-0a66-4656-8fa0-391c284c7c99" (UID: "eb611906-0a66-4656-8fa0-391c284c7c99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:44:39.817261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.817226    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb611906-0a66-4656-8fa0-391c284c7c99-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:39.817261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.817255    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcs4d\" (UniqueName: \"kubernetes.io/projected/eb611906-0a66-4656-8fa0-391c284c7c99-kube-api-access-kcs4d\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:39.989150 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.989066    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"]
Nov 02 01:44:39.992319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.992254    4851 cpu_manager.go:414] "containerMap: removing container" podUID="90e74772-510a-4359-9d31-12f064d2258b" containerName="revoke-keys"
Nov 02 01:44:39.992524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.992500    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e74772-510a-4359-9d31-12f064d2258b" containerName="revoke-keys"
Nov 02 01:44:39.992664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.992548    4851 cpu_manager.go:414] "containerMap: removing container" podUID="eb611906-0a66-4656-8fa0-391c284c7c99" containerName="revoke-keys"
Nov 02 01:44:39.992664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.992556    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb611906-0a66-4656-8fa0-391c284c7c99" containerName="revoke-keys"
Nov 02 01:44:39.992813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.992801    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e74772-510a-4359-9d31-12f064d2258b" containerName="revoke-keys"
Nov 02 01:44:39.992864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.992818    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb611906-0a66-4656-8fa0-391c284c7c99" containerName="revoke-keys"
Nov 02 01:44:39.997869 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:39.997844    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.001341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.001266    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"]
Nov 02 01:44:40.001908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.001885    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0f32d6-openshift-default-dockercfg-bh7jl\""
Nov 02 01:44:40.138313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.126544    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-certs\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.142316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.138701    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-podinfo\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.160324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.146401    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-token\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.160663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.160629    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-credential-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.160964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.160942    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istiod-ca-cert\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.161085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.161070    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-data\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.172313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.161227    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np954\" (UniqueName: \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-kube-api-access-np954\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.172595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.172570    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-envoy\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.172769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.172752    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.275376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.275263    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np954\" (UniqueName: \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-kube-api-access-np954\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.275662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.275638    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-envoy\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.275828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.275813    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.275971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.275958    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-certs\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.276170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.276156    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-podinfo\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.276321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.276306    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-token\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.276460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.276445    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-credential-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.276697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.276681    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istiod-ca-cert\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.276809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.276796    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-data\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.277496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.277470    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-data\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.296748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.296712    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-credential-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.297728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.297700    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istiod-ca-cert\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.298205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.298169    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.298656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.298633    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-certs\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.308938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.302476    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-podinfo\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.308938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.306192    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np954\" (UniqueName: \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-kube-api-access-np954\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.320407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.320370    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-token\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.340433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.332648    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-envoy\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.420680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.420651    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"
Nov 02 01:44:40.421159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.420688    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5" event={"ID":"eb611906-0a66-4656-8fa0-391c284c7c99","Type":"ContainerDied","Data":"1700e23e45de9c105801fc56437ee76ca3fdcc512c24d183d0498e8c1c3b3657"}
Nov 02 01:44:40.421159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.420750    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1700e23e45de9c105801fc56437ee76ca3fdcc512c24d183d0498e8c1c3b3657"
Nov 02 01:44:40.621239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.621152    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:40.921350 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:40.921277    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd486ac0b_a945_45aa_b47b_9ece667cb1f1.slice/crio-dd99e36d16eb647473b1cade3f431f6339259c01544bccdcee0d4091c4f4e00a WatchSource:0}: Error finding container dd99e36d16eb647473b1cade3f431f6339259c01544bccdcee0d4091c4f4e00a: Status 404 returned error can't find the container with id dd99e36d16eb647473b1cade3f431f6339259c01544bccdcee0d4091c4f4e00a
Nov 02 01:44:40.922908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.922882    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"]
Nov 02 01:44:40.924700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.924667    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:40.924792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.924744    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:40.924866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:40.924788    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:41.432324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:41.432260    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" event={"ID":"d486ac0b-a945-45aa-b47b-9ece667cb1f1","Type":"ContainerStarted","Data":"4c4f2f98196832046611760edc552add6b53e824ea6b039127289953dbcdfbe7"}
Nov 02 01:44:41.432760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:41.432363    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" event={"ID":"d486ac0b-a945-45aa-b47b-9ece667cb1f1","Type":"ContainerStarted","Data":"dd99e36d16eb647473b1cade3f431f6339259c01544bccdcee0d4091c4f4e00a"}
Nov 02 01:44:41.457097 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:41.457034    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" podStartSLOduration=2.457017755 podStartE2EDuration="2.457017755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:41.45386815 +0000 UTC m=+1786.758800978" watchObservedRunningTime="2026-11-02 01:44:41.457017755 +0000 UTC m=+1786.761950581"
Nov 02 01:44:41.628319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:41.625877    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:41.716009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:41.715906    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc" podUID="41550ac8-635a-4162-bbce-4145804093be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:44:42.627762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:42.627731    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:43.451078 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:43.451042    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:43.452104 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:43.452082    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:44:45.018710 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.018678    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"]
Nov 02 01:44:45.024559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.024521    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:44:45.037467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.037402    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"]
Nov 02 01:44:45.071427 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.071387    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-0f32d6-dockercfg-wnkqt\""
Nov 02 01:44:45.120315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.114550    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-0f32d6\""
Nov 02 01:44:45.241315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.230038    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9\" (UID: \"f66f9018-58ee-4d09-9ed7-0e20a4b6700c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:44:45.241315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.230173    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpq4\" (UniqueName: \"kubernetes.io/projected/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-kube-api-access-vzpq4\") pod \"payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9\" (UID: \"f66f9018-58ee-4d09-9ed7-0e20a4b6700c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:44:45.330971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.307330    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"]
Nov 02 01:44:45.347624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.342351    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpq4\" (UniqueName: \"kubernetes.io/projected/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-kube-api-access-vzpq4\") pod \"payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9\" (UID: \"f66f9018-58ee-4d09-9ed7-0e20a4b6700c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:44:45.347624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.343063    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9\" (UID: \"f66f9018-58ee-4d09-9ed7-0e20a4b6700c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:44:45.348508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.348480    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9\" (UID: \"f66f9018-58ee-4d09-9ed7-0e20a4b6700c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:44:45.358333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.357959    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"]
Nov 02 01:44:45.385586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.384969    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpq4\" (UniqueName: \"kubernetes.io/projected/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-kube-api-access-vzpq4\") pod \"payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9\" (UID: \"f66f9018-58ee-4d09-9ed7-0e20a4b6700c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:44:45.651585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.651502    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:44:45.954144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:45.954112    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"]
Nov 02 01:44:45.954617 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:45.954589    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66f9018_58ee_4d09_9ed7_0e20a4b6700c.slice/crio-1364f4c0296c96b7a86aae72dbd85b74080f59ce5befb9e427cfd7293ca2c792 WatchSource:0}: Error finding container 1364f4c0296c96b7a86aae72dbd85b74080f59ce5befb9e427cfd7293ca2c792: Status 404 returned error can't find the container with id 1364f4c0296c96b7a86aae72dbd85b74080f59ce5befb9e427cfd7293ca2c792
Nov 02 01:44:46.496090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:46.496052    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9" event={"ID":"f66f9018-58ee-4d09-9ed7-0e20a4b6700c","Type":"ContainerStarted","Data":"5970461acdf68562a4ca5f189f90dbe0d10daae4314e661479e4dbc81b59380c"}
Nov 02 01:44:46.496495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:46.496116    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9" event={"ID":"f66f9018-58ee-4d09-9ed7-0e20a4b6700c","Type":"ContainerStarted","Data":"1364f4c0296c96b7a86aae72dbd85b74080f59ce5befb9e427cfd7293ca2c792"}
Nov 02 01:44:46.496495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:46.496143    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:44:46.521419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:46.521362    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9" podStartSLOduration=2.521344418 podStartE2EDuration="2.521344418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:46.514692921 +0000 UTC m=+1791.819625751" watchObservedRunningTime="2026-11-02 01:44:46.521344418 +0000 UTC m=+1791.826277245"
Nov 02 01:44:47.166269 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.166233    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:44:47.233524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.233492    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41550ac8-635a-4162-bbce-4145804093be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41550ac8-635a-4162-bbce-4145804093be-plugins-config-volume\") pod \"41550ac8-635a-4162-bbce-4145804093be\" (UID: \"41550ac8-635a-4162-bbce-4145804093be\") "
Nov 02 01:44:47.233713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.233695    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41550ac8-635a-4162-bbce-4145804093be-kube-api-access-qb6pp\" (UniqueName: \"kubernetes.io/projected/41550ac8-635a-4162-bbce-4145804093be-kube-api-access-qb6pp\") pod \"41550ac8-635a-4162-bbce-4145804093be\" (UID: \"41550ac8-635a-4162-bbce-4145804093be\") "
Nov 02 01:44:47.233981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.233953    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41550ac8-635a-4162-bbce-4145804093be-plugins-config-volume" pod "41550ac8-635a-4162-bbce-4145804093be" (UID: "41550ac8-635a-4162-bbce-4145804093be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:47.234355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.234331    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41550ac8-635a-4162-bbce-4145804093be-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:47.236136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.236108    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41550ac8-635a-4162-bbce-4145804093be-kube-api-access-qb6pp" pod "41550ac8-635a-4162-bbce-4145804093be" (UID: "41550ac8-635a-4162-bbce-4145804093be"). InnerVolumeSpecName "kube-api-access-qb6pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:47.335767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.335722    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb6pp\" (UniqueName: \"kubernetes.io/projected/41550ac8-635a-4162-bbce-4145804093be-kube-api-access-qb6pp\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:44:47.506522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.506487    4851 generic.go:356] "Generic (PLEG): container finished" podID="41550ac8-635a-4162-bbce-4145804093be" containerID="715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c" exitCode=0
Nov 02 01:44:47.507013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.506586    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"
Nov 02 01:44:47.507013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.506624    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc" event={"ID":"41550ac8-635a-4162-bbce-4145804093be","Type":"ContainerDied","Data":"715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c"}
Nov 02 01:44:47.507013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.506682    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc" event={"ID":"41550ac8-635a-4162-bbce-4145804093be","Type":"ContainerDied","Data":"f88ec6b9665156be67b43e98478b5c875725f83561285281a44b0588b0ea8217"}
Nov 02 01:44:47.507013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.506706    4851 scope.go:122] "RemoveContainer" containerID="715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c"
Nov 02 01:44:47.520453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.520433    4851 scope.go:122] "RemoveContainer" containerID="715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c"
Nov 02 01:44:47.520763 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:47.520738    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c\": container with ID starting with 715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c not found: ID does not exist" containerID="715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c"
Nov 02 01:44:47.520860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.520770    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c"} err="failed to get container status \"715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c\": rpc error: code = NotFound desc = could not find container \"715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c\": container with ID starting with 715c849e3c3e0d60ca46e2d031abdc039b7dbf3a5d2978e321a1e0236d61bb2c not found: ID does not exist"
Nov 02 01:44:47.533661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.533636    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"]
Nov 02 01:44:47.547882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:47.541212    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b6ec07-567f6dd74c-5snsc"]
Nov 02 01:44:47.872419 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:44:47.869688    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:44:48.901330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:48.897244    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41550ac8-635a-4162-bbce-4145804093be" path="/var/lib/kubelet/pods/41550ac8-635a-4162-bbce-4145804093be/volumes"
Nov 02 01:44:50.308131 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:50.308074    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds" podUID="578eb675-6e20-4ed1-a546-a0bb01f3f899" containerName="payload-processing" containerID="cri-o://44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f" gracePeriod=25
Nov 02 01:44:50.358322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:50.358242    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5" podUID="2ec4c675-7706-4edf-808a-a542b0986037" containerName="payload-pre-processing" containerID="cri-o://b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001" gracePeriod=25
Nov 02 01:44:51.604532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:51.604502    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"]
Nov 02 01:44:51.608541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:51.608518    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b6ec07-2g48m"]
Nov 02 01:44:52.848251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:52.848212    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1f6375-0a3f-4053-8398-289a711be8ac" path="/var/lib/kubelet/pods/9b1f6375-0a3f-4053-8398-289a711be8ac/volumes"
Nov 02 01:44:54.783181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:54.783140    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5" podUID="2ec4c675-7706-4edf-808a-a542b0986037" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:44:54.783633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:54.783190    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds" podUID="578eb675-6e20-4ed1-a546-a0bb01f3f899" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:44:56.018799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:56.018756    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"]
Nov 02 01:44:56.035738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:56.035691    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-41215f-h8bpg"]
Nov 02 01:44:56.335998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:56.335968    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:44:56.337546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:56.337524    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:44:56.850498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:56.850462    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e74772-510a-4359-9d31-12f064d2258b" path="/var/lib/kubelet/pods/90e74772-510a-4359-9d31-12f064d2258b/volumes"
Nov 02 01:44:57.510564 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:57.510533    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:44:57.866045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:57.865955    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"]
Nov 02 01:44:57.866782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:57.866764    4851 cpu_manager.go:414] "containerMap: removing container" podUID="41550ac8-635a-4162-bbce-4145804093be" containerName="payload-pre-processing"
Nov 02 01:44:57.866854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:57.866783    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41550ac8-635a-4162-bbce-4145804093be" containerName="payload-pre-processing"
Nov 02 01:44:57.866954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:57.866943    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="41550ac8-635a-4162-bbce-4145804093be" containerName="payload-pre-processing"
Nov 02 01:44:57.872416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:57.872390    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:57.878403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:57.878374    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:44:57.878580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:57.878451    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:44:57.880874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:57.880847    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"]
Nov 02 01:44:58.005946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.005914    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd4962e9-9hqh9\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:58.006143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.006064    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd4962e9-9hqh9\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:58.006143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.006093    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx7qx\" (UniqueName: \"kubernetes.io/projected/b96ed607-e467-4e00-86af-ac6f44a46e54-kube-api-access-rx7qx\") pod \"maas-api-revoke-sub-cd4962e9-9hqh9\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:58.107890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.107852    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd4962e9-9hqh9\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:58.107890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.107898    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx7qx\" (UniqueName: \"kubernetes.io/projected/b96ed607-e467-4e00-86af-ac6f44a46e54-kube-api-access-rx7qx\") pod \"maas-api-revoke-sub-cd4962e9-9hqh9\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:58.108154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.108054    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd4962e9-9hqh9\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:58.108573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.108552    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd4962e9-9hqh9\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:58.110932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.110907    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd4962e9-9hqh9\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:58.120101 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.120040    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx7qx\" (UniqueName: \"kubernetes.io/projected/b96ed607-e467-4e00-86af-ac6f44a46e54-kube-api-access-rx7qx\") pod \"maas-api-revoke-sub-cd4962e9-9hqh9\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:58.192597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.192252    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:44:58.552421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.552394    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"]
Nov 02 01:44:58.553544 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:44:58.553517    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96ed607_e467_4e00_86af_ac6f44a46e54.slice/crio-5e4e8d7c8061184f5e11853c78fac1a9927595fe49dd7c36c7a2530016036dea WatchSource:0}: Error finding container 5e4e8d7c8061184f5e11853c78fac1a9927595fe49dd7c36c7a2530016036dea: Status 404 returned error can't find the container with id 5e4e8d7c8061184f5e11853c78fac1a9927595fe49dd7c36c7a2530016036dea
Nov 02 01:44:58.891105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:58.891064    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9" event={"ID":"b96ed607-e467-4e00-86af-ac6f44a46e54","Type":"ContainerStarted","Data":"5e4e8d7c8061184f5e11853c78fac1a9927595fe49dd7c36c7a2530016036dea"}
Nov 02 01:44:59.374320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:59.369537    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"]
Nov 02 01:44:59.374320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:59.369826    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy" containerID="cri-o://c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24" gracePeriod=30
Nov 02 01:44:59.909133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:59.909089    4851 generic.go:356] "Generic (PLEG): container finished" podID="b96ed607-e467-4e00-86af-ac6f44a46e54" containerID="e41050565b293c3c2ccd180631c8011edcf24c362c2768fdc58c90937a1d4060" exitCode=0
Nov 02 01:44:59.909660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:44:59.909182    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9" event={"ID":"b96ed607-e467-4e00-86af-ac6f44a46e54","Type":"ContainerDied","Data":"e41050565b293c3c2ccd180631c8011edcf24c362c2768fdc58c90937a1d4060"}
Nov 02 01:45:00.154848 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.154758    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"]
Nov 02 01:45:00.159553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.159525    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"
Nov 02 01:45:00.161272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.161244    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"]
Nov 02 01:45:00.174156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.174126    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"]
Nov 02 01:45:00.174426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.174400    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"
Nov 02 01:45:00.180645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.180605    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"]
Nov 02 01:45:00.180820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.180704    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"
Nov 02 01:45:00.191315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.190435    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"]
Nov 02 01:45:00.254471 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.254357    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"]
Nov 02 01:45:00.274537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.273871    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt\" (UID: \"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"
Nov 02 01:45:00.274537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.274047    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk\" (UID: \"93ffddad-f58a-4f2a-8eca-0a95c5ac330d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"
Nov 02 01:45:00.274537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.274149    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893065-bst7j\" (UID: \"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"
Nov 02 01:45:00.274537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.274187    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4t6x\" (UniqueName: \"kubernetes.io/projected/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-kube-api-access-d4t6x\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk\" (UID: \"93ffddad-f58a-4f2a-8eca-0a95c5ac330d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"
Nov 02 01:45:00.274537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.274215    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwkpw\" (UniqueName: \"kubernetes.io/projected/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-kube-api-access-kwkpw\") pod \"maas-api-key-cleanup-29893065-bst7j\" (UID: \"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"
Nov 02 01:45:00.274537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.274333    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrfg\" (UniqueName: \"kubernetes.io/projected/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-kube-api-access-rwrfg\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt\" (UID: \"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"
Nov 02 01:45:00.319948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.319910    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"]
Nov 02 01:45:00.322138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.322109    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:00.326714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.326687    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"
Nov 02 01:45:00.333411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.333384    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"]
Nov 02 01:45:00.352896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.352864    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"]
Nov 02 01:45:00.357909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.357881    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"]
Nov 02 01:45:00.365887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.365858    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"]
Nov 02 01:45:00.371906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.371880    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"
Nov 02 01:45:00.373431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.373403    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"]
Nov 02 01:45:00.376311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.376266    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrfg\" (UniqueName: \"kubernetes.io/projected/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-kube-api-access-rwrfg\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt\" (UID: \"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"
Nov 02 01:45:00.376454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.376432    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt\" (UID: \"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"
Nov 02 01:45:00.376528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.376506    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfshq\" (UniqueName: \"kubernetes.io/projected/119a2d02-b88d-4052-b81c-a43a441ff466-kube-api-access-rfshq\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn\" (UID: \"119a2d02-b88d-4052-b81c-a43a441ff466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:00.376630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.376611    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvwl\" (UniqueName: \"kubernetes.io/projected/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-kube-api-access-vbvwl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng\" (UID: \"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"
Nov 02 01:45:00.376691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.376658    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk\" (UID: \"93ffddad-f58a-4f2a-8eca-0a95c5ac330d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"
Nov 02 01:45:00.376769 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.376753    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/119a2d02-b88d-4052-b81c-a43a441ff466-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn\" (UID: \"119a2d02-b88d-4052-b81c-a43a441ff466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:00.376826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.376795    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng\" (UID: \"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"
Nov 02 01:45:00.376881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.376827    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893065-bst7j\" (UID: \"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"
Nov 02 01:45:00.376881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.376865    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4t6x\" (UniqueName: \"kubernetes.io/projected/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-kube-api-access-d4t6x\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk\" (UID: \"93ffddad-f58a-4f2a-8eca-0a95c5ac330d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"
Nov 02 01:45:00.376979 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.376894    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwkpw\" (UniqueName: \"kubernetes.io/projected/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-kube-api-access-kwkpw\") pod \"maas-api-key-cleanup-29893065-bst7j\" (UID: \"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"
Nov 02 01:45:00.380379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.378816    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk\" (UID: \"93ffddad-f58a-4f2a-8eca-0a95c5ac330d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"
Nov 02 01:45:00.380379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.379459    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt\" (UID: \"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"
Nov 02 01:45:00.380379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.380116    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893065-bst7j\" (UID: \"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"
Nov 02 01:45:00.380598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.380435    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"
Nov 02 01:45:00.382227 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.382207    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"]
Nov 02 01:45:00.389419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.389385    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"
Nov 02 01:45:00.390896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.390869    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"]
Nov 02 01:45:00.392306 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.392264    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwkpw\" (UniqueName: \"kubernetes.io/projected/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-kube-api-access-kwkpw\") pod \"maas-api-key-cleanup-29893065-bst7j\" (UID: \"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"
Nov 02 01:45:00.398154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.398130    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4t6x\" (UniqueName: \"kubernetes.io/projected/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-kube-api-access-d4t6x\") pod \"maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk\" (UID: \"93ffddad-f58a-4f2a-8eca-0a95c5ac330d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"
Nov 02 01:45:00.398274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.398248    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"]
Nov 02 01:45:00.401242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.401215    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrfg\" (UniqueName: \"kubernetes.io/projected/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-kube-api-access-rwrfg\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt\" (UID: \"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"
Nov 02 01:45:00.405731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.405682    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"]
Nov 02 01:45:00.478775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.478734    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9x5t\" (UniqueName: \"kubernetes.io/projected/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-kube-api-access-k9x5t\") pod \"maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh\" (UID: \"26f9bef1-c9ff-4de5-82e1-43eb7a1709da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"
Nov 02 01:45:00.478970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.478882    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnsg\" (UniqueName: \"kubernetes.io/projected/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-kube-api-access-qcnsg\") pod \"maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m\" (UID: \"0d49aac2-82f1-48cf-91c9-07298ba7b4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"
Nov 02 01:45:00.478970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.478923    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh\" (UID: \"26f9bef1-c9ff-4de5-82e1-43eb7a1709da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"
Nov 02 01:45:00.478970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.478956    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfshq\" (UniqueName: \"kubernetes.io/projected/119a2d02-b88d-4052-b81c-a43a441ff466-kube-api-access-rfshq\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn\" (UID: \"119a2d02-b88d-4052-b81c-a43a441ff466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:00.479143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.479073    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvwl\" (UniqueName: \"kubernetes.io/projected/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-kube-api-access-vbvwl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng\" (UID: \"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"
Nov 02 01:45:00.479201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.479147    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjx9d\" (UniqueName: \"kubernetes.io/projected/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-kube-api-access-kjx9d\") pod \"maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x\" (UID: \"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"
Nov 02 01:45:00.479253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.479204    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/119a2d02-b88d-4052-b81c-a43a441ff466-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn\" (UID: \"119a2d02-b88d-4052-b81c-a43a441ff466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:00.479328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.479284    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng\" (UID: \"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"
Nov 02 01:45:00.479455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.479437    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x\" (UID: \"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"
Nov 02 01:45:00.479537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.479501    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m\" (UID: \"0d49aac2-82f1-48cf-91c9-07298ba7b4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"
Nov 02 01:45:00.480080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.480055    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng\" (UID: \"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"
Nov 02 01:45:00.480177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.480056    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/119a2d02-b88d-4052-b81c-a43a441ff466-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn\" (UID: \"119a2d02-b88d-4052-b81c-a43a441ff466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:00.492347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.492282    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfshq\" (UniqueName: \"kubernetes.io/projected/119a2d02-b88d-4052-b81c-a43a441ff466-kube-api-access-rfshq\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn\" (UID: \"119a2d02-b88d-4052-b81c-a43a441ff466\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:00.492470 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.492282    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvwl\" (UniqueName: \"kubernetes.io/projected/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-kube-api-access-vbvwl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng\" (UID: \"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"
Nov 02 01:45:00.581384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.581349    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9x5t\" (UniqueName: \"kubernetes.io/projected/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-kube-api-access-k9x5t\") pod \"maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh\" (UID: \"26f9bef1-c9ff-4de5-82e1-43eb7a1709da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"
Nov 02 01:45:00.581573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.581462    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnsg\" (UniqueName: \"kubernetes.io/projected/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-kube-api-access-qcnsg\") pod \"maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m\" (UID: \"0d49aac2-82f1-48cf-91c9-07298ba7b4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"
Nov 02 01:45:00.581573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.581499    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh\" (UID: \"26f9bef1-c9ff-4de5-82e1-43eb7a1709da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"
Nov 02 01:45:00.581678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.581653    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjx9d\" (UniqueName: \"kubernetes.io/projected/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-kube-api-access-kjx9d\") pod \"maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x\" (UID: \"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"
Nov 02 01:45:00.581767 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.581750    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x\" (UID: \"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"
Nov 02 01:45:00.581860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.581840    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m\" (UID: \"0d49aac2-82f1-48cf-91c9-07298ba7b4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"
Nov 02 01:45:00.582398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.582379    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x\" (UID: \"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"
Nov 02 01:45:00.582450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.582384    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh\" (UID: \"26f9bef1-c9ff-4de5-82e1-43eb7a1709da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"
Nov 02 01:45:00.582527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.582508    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m\" (UID: \"0d49aac2-82f1-48cf-91c9-07298ba7b4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"
Nov 02 01:45:00.594228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.594188    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9x5t\" (UniqueName: \"kubernetes.io/projected/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-kube-api-access-k9x5t\") pod \"maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh\" (UID: \"26f9bef1-c9ff-4de5-82e1-43eb7a1709da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"
Nov 02 01:45:00.594377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.594234    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnsg\" (UniqueName: \"kubernetes.io/projected/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-kube-api-access-qcnsg\") pod \"maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m\" (UID: \"0d49aac2-82f1-48cf-91c9-07298ba7b4c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"
Nov 02 01:45:00.594377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.594233    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjx9d\" (UniqueName: \"kubernetes.io/projected/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-kube-api-access-kjx9d\") pod \"maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x\" (UID: \"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"
Nov 02 01:45:00.604105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.604079    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"
Nov 02 01:45:00.636076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.636046    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"
Nov 02 01:45:00.657568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.657511    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"
Nov 02 01:45:00.672747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.669684    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:00.687029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.687005    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"
Nov 02 01:45:00.699593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.699565    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"
Nov 02 01:45:00.739025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.738987    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"
Nov 02 01:45:00.751979 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:00.750521    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"
Nov 02 01:45:01.021766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.018863    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"]
Nov 02 01:45:01.229671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.229454    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"]
Nov 02 01:45:01.234603 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:01.234517    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ffddad_f58a_4f2a_8eca_0a95c5ac330d.slice/crio-0e31ec99de8ca9794d9d2c0c4c82b168fedd6b2214de8b0a5a48ac579723453e WatchSource:0}: Error finding container 0e31ec99de8ca9794d9d2c0c4c82b168fedd6b2214de8b0a5a48ac579723453e: Status 404 returned error can't find the container with id 0e31ec99de8ca9794d9d2c0c4c82b168fedd6b2214de8b0a5a48ac579723453e
Nov 02 01:45:01.239595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.239235    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"]
Nov 02 01:45:01.246320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.246273    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"]
Nov 02 01:45:01.258947 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:01.258913    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119a2d02_b88d_4052_b81c_a43a441ff466.slice/crio-d2ac6d230e52d04c7d9a0e53cd5a7c1331b063b7fdb2c9fac86ff73cf7ee0b42 WatchSource:0}: Error finding container d2ac6d230e52d04c7d9a0e53cd5a7c1331b063b7fdb2c9fac86ff73cf7ee0b42: Status 404 returned error can't find the container with id d2ac6d230e52d04c7d9a0e53cd5a7c1331b063b7fdb2c9fac86ff73cf7ee0b42
Nov 02 01:45:01.379457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.379418    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"]
Nov 02 01:45:01.410419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.410387    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"]
Nov 02 01:45:01.425301 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.425254    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"]
Nov 02 01:45:01.440042 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:01.440008    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd965b026_5f19_49bd_9cd4_9c8e2c16ecc0.slice/crio-228399f5e9823fcecd711f9e4a22e3674ca864fcae7efc1a6379f333a09ecb8c WatchSource:0}: Error finding container 228399f5e9823fcecd711f9e4a22e3674ca864fcae7efc1a6379f333a09ecb8c: Status 404 returned error can't find the container with id 228399f5e9823fcecd711f9e4a22e3674ca864fcae7efc1a6379f333a09ecb8c
Nov 02 01:45:01.461943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.461913    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"]
Nov 02 01:45:01.535381 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:01.535270    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9587a258_f1ba_4ce4_bba5_1c0e4ef8cd45.slice/crio-8b5faf4e62a78cadb117592c8299781fb586315d81ea3896754e8bc70dc707c3 WatchSource:0}: Error finding container 8b5faf4e62a78cadb117592c8299781fb586315d81ea3896754e8bc70dc707c3: Status 404 returned error can't find the container with id 8b5faf4e62a78cadb117592c8299781fb586315d81ea3896754e8bc70dc707c3
Nov 02 01:45:01.536457 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:01.536434    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d49aac2_82f1_48cf_91c9_07298ba7b4c3.slice/crio-3598772b02b4a71547999a702b528ec0bc9d3be07d1abdb4bba24181565cb403 WatchSource:0}: Error finding container 3598772b02b4a71547999a702b528ec0bc9d3be07d1abdb4bba24181565cb403: Status 404 returned error can't find the container with id 3598772b02b4a71547999a702b528ec0bc9d3be07d1abdb4bba24181565cb403
Nov 02 01:45:01.730741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.730709    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:45:01.841245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.841160    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96ed607-e467-4e00-86af-ac6f44a46e54-kube-api-access-rx7qx\" (UniqueName: \"kubernetes.io/projected/b96ed607-e467-4e00-86af-ac6f44a46e54-kube-api-access-rx7qx\") pod \"b96ed607-e467-4e00-86af-ac6f44a46e54\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") "
Nov 02 01:45:01.841457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.841372    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-xks-ca\") pod \"b96ed607-e467-4e00-86af-ac6f44a46e54\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") "
Nov 02 01:45:01.841457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.841409    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-openshift-ca\") pod \"b96ed607-e467-4e00-86af-ac6f44a46e54\" (UID: \"b96ed607-e467-4e00-86af-ac6f44a46e54\") "
Nov 02 01:45:01.841824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.841801    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-openshift-ca" pod "b96ed607-e467-4e00-86af-ac6f44a46e54" (UID: "b96ed607-e467-4e00-86af-ac6f44a46e54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:01.842051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.842031    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:01.936745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.936711    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-xks-ca" pod "b96ed607-e467-4e00-86af-ac6f44a46e54" (UID: "b96ed607-e467-4e00-86af-ac6f44a46e54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:45:01.937712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.937681    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96ed607-e467-4e00-86af-ac6f44a46e54-kube-api-access-rx7qx" pod "b96ed607-e467-4e00-86af-ac6f44a46e54" (UID: "b96ed607-e467-4e00-86af-ac6f44a46e54"). InnerVolumeSpecName "kube-api-access-rx7qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:01.943838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.943805    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96ed607-e467-4e00-86af-ac6f44a46e54-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:01.943838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:01.943836    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx7qx\" (UniqueName: \"kubernetes.io/projected/b96ed607-e467-4e00-86af-ac6f44a46e54-kube-api-access-rx7qx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:02.060610 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.060566    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh" event={"ID":"26f9bef1-c9ff-4de5-82e1-43eb7a1709da","Type":"ContainerStarted","Data":"117d4e15252880aa56c9ebd025ed1893db7f89fca094aa92dc1fe610e7d7aabd"}
Nov 02 01:45:02.062080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.062061    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"
Nov 02 01:45:02.062229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.062056    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9" event={"ID":"b96ed607-e467-4e00-86af-ac6f44a46e54","Type":"ContainerDied","Data":"5e4e8d7c8061184f5e11853c78fac1a9927595fe49dd7c36c7a2530016036dea"}
Nov 02 01:45:02.062229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.062188    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e4e8d7c8061184f5e11853c78fac1a9927595fe49dd7c36c7a2530016036dea"
Nov 02 01:45:02.064998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.064623    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x" event={"ID":"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45","Type":"ContainerStarted","Data":"8b5faf4e62a78cadb117592c8299781fb586315d81ea3896754e8bc70dc707c3"}
Nov 02 01:45:02.068658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.068630    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" event={"ID":"93ffddad-f58a-4f2a-8eca-0a95c5ac330d","Type":"ContainerStarted","Data":"79e9ef96ece48a1f1fae061a6c0048efdac5b75423e2e0ecf441570e3ef3a5fc"}
Nov 02 01:45:02.068825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.068798    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" event={"ID":"93ffddad-f58a-4f2a-8eca-0a95c5ac330d","Type":"ContainerStarted","Data":"0e31ec99de8ca9794d9d2c0c4c82b168fedd6b2214de8b0a5a48ac579723453e"}
Nov 02 01:45:02.071450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.071397    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng" event={"ID":"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0","Type":"ContainerStarted","Data":"228399f5e9823fcecd711f9e4a22e3674ca864fcae7efc1a6379f333a09ecb8c"}
Nov 02 01:45:02.073902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.073757    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" event={"ID":"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134","Type":"ContainerStarted","Data":"8b68bedf4f2a413beb81cf8bd2b98702f56f748f6d2fb9eaba8fb61be6f7d8b2"}
Nov 02 01:45:02.073902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.073807    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" event={"ID":"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134","Type":"ContainerStarted","Data":"2d509ec4ac26fe25251bed2423551c8bd0b28e5310c475fb5b5d8c906812ddf6"}
Nov 02 01:45:02.075825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.075801    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" event={"ID":"0d49aac2-82f1-48cf-91c9-07298ba7b4c3","Type":"ContainerStarted","Data":"3598772b02b4a71547999a702b528ec0bc9d3be07d1abdb4bba24181565cb403"}
Nov 02 01:45:02.077616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.077590    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" event={"ID":"119a2d02-b88d-4052-b81c-a43a441ff466","Type":"ContainerStarted","Data":"aa6da8765d787061103dc21913c4751cdcb2baded1412c26d66124955f974abd"}
Nov 02 01:45:02.077709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.077638    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" event={"ID":"119a2d02-b88d-4052-b81c-a43a441ff466","Type":"ContainerStarted","Data":"d2ac6d230e52d04c7d9a0e53cd5a7c1331b063b7fdb2c9fac86ff73cf7ee0b42"}
Nov 02 01:45:02.079707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.079687    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" event={"ID":"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab","Type":"ContainerStarted","Data":"640a9cf67206b5f94ad8558d0cc0154ea814d2f52cda1bc0741f4175ea43d146"}
Nov 02 01:45:02.079815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.079729    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" event={"ID":"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab","Type":"ContainerStarted","Data":"c462459e29edf7f3cae53a89ec83b98e470a4f475b56eaffc7113b7874a285c2"}
Nov 02 01:45:02.095137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.095049    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" podStartSLOduration=2.095035269 podStartE2EDuration="2.095035269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:02.092518662 +0000 UTC m=+1807.397451481" watchObservedRunningTime="2026-11-02 01:45:02.095035269 +0000 UTC m=+1807.399968111"
Nov 02 01:45:02.106588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.106555    4851 scope.go:122] "RemoveContainer" containerID="2b681c18b956eaa5c0a9db4063765ff4564890e1b9e1ae9726686a31d891e5df"
Nov 02 01:45:02.110997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.110944    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" podStartSLOduration=2.110929573 podStartE2EDuration="2.110929573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:02.11010494 +0000 UTC m=+1807.415037791" watchObservedRunningTime="2026-11-02 01:45:02.110929573 +0000 UTC m=+1807.415862400"
Nov 02 01:45:02.120392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.120369    4851 scope.go:122] "RemoveContainer" containerID="9da8e1663b38c77998abbb6a53c11a842ef0de9236786cb8cbbf97a363749fc9"
Nov 02 01:45:02.217537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.217477    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" podStartSLOduration=2.217461309 podStartE2EDuration="2.217461309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:02.156781219 +0000 UTC m=+1807.461714046" watchObservedRunningTime="2026-11-02 01:45:02.217461309 +0000 UTC m=+1807.522394136"
Nov 02 01:45:02.219662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:02.219617    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" podStartSLOduration=2.219607416 podStartE2EDuration="2.219607416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:02.204512599 +0000 UTC m=+1807.509445428" watchObservedRunningTime="2026-11-02 01:45:02.219607416 +0000 UTC m=+1807.524540254"
Nov 02 01:45:02.904842 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:02.904771    4851 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f9bef1_c9ff_4de5_82e1_43eb7a1709da.slice/crio-a6bd784f3c92a9c9192ea2a23460367bbc59cf9266beb208994e184930e96e1f.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 01:45:03.116253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.114771    4851 generic.go:356] "Generic (PLEG): container finished" podID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerID="6a6fa11caf40f5f5ba45fd742b5d9ff76bbbbef313239c8eadf24008c1e8a482" exitCode=60
Nov 02 01:45:03.116253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.114915    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" event={"ID":"0d49aac2-82f1-48cf-91c9-07298ba7b4c3","Type":"ContainerDied","Data":"6a6fa11caf40f5f5ba45fd742b5d9ff76bbbbef313239c8eadf24008c1e8a482"}
Nov 02 01:45:03.116253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.115392    4851 scope.go:122] "RemoveContainer" containerID="6a6fa11caf40f5f5ba45fd742b5d9ff76bbbbef313239c8eadf24008c1e8a482"
Nov 02 01:45:03.138977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.135498    4851 generic.go:356] "Generic (PLEG): container finished" podID="119a2d02-b88d-4052-b81c-a43a441ff466" containerID="aa6da8765d787061103dc21913c4751cdcb2baded1412c26d66124955f974abd" exitCode=60
Nov 02 01:45:03.138977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.135771    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" event={"ID":"119a2d02-b88d-4052-b81c-a43a441ff466","Type":"ContainerDied","Data":"aa6da8765d787061103dc21913c4751cdcb2baded1412c26d66124955f974abd"}
Nov 02 01:45:03.138977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.136232    4851 scope.go:122] "RemoveContainer" containerID="aa6da8765d787061103dc21913c4751cdcb2baded1412c26d66124955f974abd"
Nov 02 01:45:03.139448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.139425    4851 generic.go:356] "Generic (PLEG): container finished" podID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerID="640a9cf67206b5f94ad8558d0cc0154ea814d2f52cda1bc0741f4175ea43d146" exitCode=60
Nov 02 01:45:03.140094 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.139925    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" event={"ID":"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab","Type":"ContainerDied","Data":"640a9cf67206b5f94ad8558d0cc0154ea814d2f52cda1bc0741f4175ea43d146"}
Nov 02 01:45:03.140437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.140418    4851 scope.go:122] "RemoveContainer" containerID="640a9cf67206b5f94ad8558d0cc0154ea814d2f52cda1bc0741f4175ea43d146"
Nov 02 01:45:03.144085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.144062    4851 generic.go:356] "Generic (PLEG): container finished" podID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerID="a6bd784f3c92a9c9192ea2a23460367bbc59cf9266beb208994e184930e96e1f" exitCode=60
Nov 02 01:45:03.144242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.144209    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh" event={"ID":"26f9bef1-c9ff-4de5-82e1-43eb7a1709da","Type":"ContainerDied","Data":"a6bd784f3c92a9c9192ea2a23460367bbc59cf9266beb208994e184930e96e1f"}
Nov 02 01:45:03.144533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.144519    4851 scope.go:122] "RemoveContainer" containerID="a6bd784f3c92a9c9192ea2a23460367bbc59cf9266beb208994e184930e96e1f"
Nov 02 01:45:03.146121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.146096    4851 generic.go:356] "Generic (PLEG): container finished" podID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerID="866a0510a084ceae140d1c9a6f686ee7dd505bfed7fa24a90ef48ae978f95add" exitCode=60
Nov 02 01:45:03.146231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.146215    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x" event={"ID":"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45","Type":"ContainerDied","Data":"866a0510a084ceae140d1c9a6f686ee7dd505bfed7fa24a90ef48ae978f95add"}
Nov 02 01:45:03.146679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.146409    4851 scope.go:122] "RemoveContainer" containerID="866a0510a084ceae140d1c9a6f686ee7dd505bfed7fa24a90ef48ae978f95add"
Nov 02 01:45:03.148965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.148945    4851 generic.go:356] "Generic (PLEG): container finished" podID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerID="79e9ef96ece48a1f1fae061a6c0048efdac5b75423e2e0ecf441570e3ef3a5fc" exitCode=60
Nov 02 01:45:03.149036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.149007    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" event={"ID":"93ffddad-f58a-4f2a-8eca-0a95c5ac330d","Type":"ContainerDied","Data":"79e9ef96ece48a1f1fae061a6c0048efdac5b75423e2e0ecf441570e3ef3a5fc"}
Nov 02 01:45:03.149236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.149219    4851 scope.go:122] "RemoveContainer" containerID="79e9ef96ece48a1f1fae061a6c0048efdac5b75423e2e0ecf441570e3ef3a5fc"
Nov 02 01:45:03.150721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.150699    4851 generic.go:356] "Generic (PLEG): container finished" podID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerID="d83afbb55596be7abb7ef6fa0ad532136ca7984dfeec88adfe438d24d6d5a120" exitCode=60
Nov 02 01:45:03.150831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.150772    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng" event={"ID":"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0","Type":"ContainerDied","Data":"d83afbb55596be7abb7ef6fa0ad532136ca7984dfeec88adfe438d24d6d5a120"}
Nov 02 01:45:03.150984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.150968    4851 scope.go:122] "RemoveContainer" containerID="d83afbb55596be7abb7ef6fa0ad532136ca7984dfeec88adfe438d24d6d5a120"
Nov 02 01:45:03.152431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.152409    4851 generic.go:356] "Generic (PLEG): container finished" podID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerID="8b68bedf4f2a413beb81cf8bd2b98702f56f748f6d2fb9eaba8fb61be6f7d8b2" exitCode=60
Nov 02 01:45:03.152537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.152492    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" event={"ID":"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134","Type":"ContainerDied","Data":"8b68bedf4f2a413beb81cf8bd2b98702f56f748f6d2fb9eaba8fb61be6f7d8b2"}
Nov 02 01:45:03.152679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:03.152657    4851 scope.go:122] "RemoveContainer" containerID="8b68bedf4f2a413beb81cf8bd2b98702f56f748f6d2fb9eaba8fb61be6f7d8b2"
Nov 02 01:45:04.164899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.164868    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" event={"ID":"119a2d02-b88d-4052-b81c-a43a441ff466","Type":"ContainerStarted","Data":"92cd14779073bdae9470632c847137fac0017d7266ccd1e72841308bb813d476"}
Nov 02 01:45:04.167010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.166988    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" event={"ID":"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab","Type":"ContainerStarted","Data":"03b633b463a7a934ebe9c3b133e8089cc79d1104f76e7de0bdf7d19b597cbcae"}
Nov 02 01:45:04.169185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.169160    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh" event={"ID":"26f9bef1-c9ff-4de5-82e1-43eb7a1709da","Type":"ContainerStarted","Data":"5f565e637e74f68f962c8bc9683f1fa0d04e2a1682dc26a2689d8a0e518712d9"}
Nov 02 01:45:04.171583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.171544    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x" event={"ID":"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45","Type":"ContainerStarted","Data":"4653bae9622dce45e14810e9d1928adb35ecc94fe668964b9b9b3dad0a3479c0"}
Nov 02 01:45:04.173553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.173527    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" event={"ID":"93ffddad-f58a-4f2a-8eca-0a95c5ac330d","Type":"ContainerStarted","Data":"21551b73ce361fa492dbeb429008926bf3e85ee228dbc18887210c7c3e466f51"}
Nov 02 01:45:04.175614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.175591    4851 generic.go:356] "Generic (PLEG): container finished" podID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerID="4dcbf341aa6a601ed8ada1eec9b8fa9c73101f99519a3a06d63dc1b8587a6b90" exitCode=60
Nov 02 01:45:04.175716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.175666    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng" event={"ID":"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0","Type":"ContainerDied","Data":"4dcbf341aa6a601ed8ada1eec9b8fa9c73101f99519a3a06d63dc1b8587a6b90"}
Nov 02 01:45:04.175776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.175715    4851 scope.go:122] "RemoveContainer" containerID="d83afbb55596be7abb7ef6fa0ad532136ca7984dfeec88adfe438d24d6d5a120"
Nov 02 01:45:04.175940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.175920    4851 scope.go:122] "RemoveContainer" containerID="4dcbf341aa6a601ed8ada1eec9b8fa9c73101f99519a3a06d63dc1b8587a6b90"
Nov 02 01:45:04.176218 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:04.176195    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng_odh-ai-gateway-infra(d965b026-5f19-49bd-9cd4-9c8e2c16ecc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0"
Nov 02 01:45:04.178077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.178052    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" event={"ID":"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134","Type":"ContainerStarted","Data":"5ce22cd7667a418b174e5b44573dffa3774e02529579e6452e945c048895aa5b"}
Nov 02 01:45:04.180960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.180930    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" event={"ID":"0d49aac2-82f1-48cf-91c9-07298ba7b4c3","Type":"ContainerStarted","Data":"71f294437be909815f8df3f3d42c27fa19fef6f3fda8b0b1d4c1afd7bff3ebb7"}
Nov 02 01:45:04.216684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.215544    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x" podStartSLOduration=4.215521036 podStartE2EDuration="4.215521036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:04.201340705 +0000 UTC m=+1809.506273527" watchObservedRunningTime="2026-11-02 01:45:04.215521036 +0000 UTC m=+1809.520453862"
Nov 02 01:45:04.250592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.250534    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh" podStartSLOduration=4.250513197 podStartE2EDuration="4.250513197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:04.248504391 +0000 UTC m=+1809.553437219" watchObservedRunningTime="2026-11-02 01:45:04.250513197 +0000 UTC m=+1809.555446023"
Nov 02 01:45:04.351658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.351613    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" podStartSLOduration=4.351600511 podStartE2EDuration="4.351600511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:04.346776281 +0000 UTC m=+1809.651709108" watchObservedRunningTime="2026-11-02 01:45:04.351600511 +0000 UTC m=+1809.656533337"
Nov 02 01:45:04.480679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.480645    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"]
Nov 02 01:45:04.481134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.481105    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy" containerID="cri-o://8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121" gracePeriod=30
Nov 02 01:45:04.827371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.811032    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5" podUID="2ec4c675-7706-4edf-808a-a542b0986037" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:45:04.827371 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.826924    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds" podUID="578eb675-6e20-4ed1-a546-a0bb01f3f899" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:45:04.936924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.936890    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:45:04.982919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.982879    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-credential-socket\") pod \"504edd1e-79af-4ceb-85a7-507a8f6de614\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") "
Nov 02 01:45:04.983103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.982968    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-podinfo\") pod \"504edd1e-79af-4ceb-85a7-507a8f6de614\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") "
Nov 02 01:45:04.983103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.983004    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-token\" (UniqueName: \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-token\") pod \"504edd1e-79af-4ceb-85a7-507a8f6de614\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") "
Nov 02 01:45:04.983103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.983045    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504edd1e-79af-4ceb-85a7-507a8f6de614-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504edd1e-79af-4ceb-85a7-507a8f6de614-istiod-ca-cert\") pod \"504edd1e-79af-4ceb-85a7-507a8f6de614\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") "
Nov 02 01:45:04.983103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.983086    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-envoy\") pod \"504edd1e-79af-4ceb-85a7-507a8f6de614\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") "
Nov 02 01:45:04.983393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.983214    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-socket\") pod \"504edd1e-79af-4ceb-85a7-507a8f6de614\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") "
Nov 02 01:45:04.983393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.983260    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-data\") pod \"504edd1e-79af-4ceb-85a7-507a8f6de614\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") "
Nov 02 01:45:04.983393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.983310    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-kube-api-access-nb6p6\" (UniqueName: \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-kube-api-access-nb6p6\") pod \"504edd1e-79af-4ceb-85a7-507a8f6de614\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") "
Nov 02 01:45:04.983393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.983337    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-certs\") pod \"504edd1e-79af-4ceb-85a7-507a8f6de614\" (UID: \"504edd1e-79af-4ceb-85a7-507a8f6de614\") "
Nov 02 01:45:04.984792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.984631    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-socket" pod "504edd1e-79af-4ceb-85a7-507a8f6de614" (UID: "504edd1e-79af-4ceb-85a7-507a8f6de614"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:04.984792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.984649    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504edd1e-79af-4ceb-85a7-507a8f6de614-istiod-ca-cert" pod "504edd1e-79af-4ceb-85a7-507a8f6de614" (UID: "504edd1e-79af-4ceb-85a7-507a8f6de614"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:04.985029 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.984941    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-credential-socket" pod "504edd1e-79af-4ceb-85a7-507a8f6de614" (UID: "504edd1e-79af-4ceb-85a7-507a8f6de614"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:04.985268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.985248    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-certs" pod "504edd1e-79af-4ceb-85a7-507a8f6de614" (UID: "504edd1e-79af-4ceb-85a7-507a8f6de614"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:04.985679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.985653    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-data" pod "504edd1e-79af-4ceb-85a7-507a8f6de614" (UID: "504edd1e-79af-4ceb-85a7-507a8f6de614"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:04.985902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.985814    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"]
Nov 02 01:45:04.987465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.987436    4851 cpu_manager.go:414] "containerMap: removing container" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy"
Nov 02 01:45:04.987549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.987466    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy"
Nov 02 01:45:04.987549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.987519    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b96ed607-e467-4e00-86af-ac6f44a46e54" containerName="revoke-keys"
Nov 02 01:45:04.987549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.987527    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96ed607-e467-4e00-86af-ac6f44a46e54" containerName="revoke-keys"
Nov 02 01:45:04.987805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.987789    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerName="istio-proxy"
Nov 02 01:45:04.987839 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.987819    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96ed607-e467-4e00-86af-ac6f44a46e54" containerName="revoke-keys"
Nov 02 01:45:04.991228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.991148    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-token" pod "504edd1e-79af-4ceb-85a7-507a8f6de614" (UID: "504edd1e-79af-4ceb-85a7-507a8f6de614"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:04.991367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.991234    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-podinfo" pod "504edd1e-79af-4ceb-85a7-507a8f6de614" (UID: "504edd1e-79af-4ceb-85a7-507a8f6de614"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:45:04.992700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.992664    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-envoy" pod "504edd1e-79af-4ceb-85a7-507a8f6de614" (UID: "504edd1e-79af-4ceb-85a7-507a8f6de614"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:04.995569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:04.995511    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.002000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.001972    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-kube-api-access-nb6p6" pod "504edd1e-79af-4ceb-85a7-507a8f6de614" (UID: "504edd1e-79af-4ceb-85a7-507a8f6de614"). InnerVolumeSpecName "kube-api-access-nb6p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:05.002219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.002199    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"]
Nov 02 01:45:05.035636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.035599    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:45:05.096496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.095528    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:05.096496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.095574    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:05.096496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.095592    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:05.096496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.095606    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504edd1e-79af-4ceb-85a7-507a8f6de614-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:05.096496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.095621    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:05.096496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.095634    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:05.096496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.095647    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:05.096496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.095680    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb6p6\" (UniqueName: \"kubernetes.io/projected/504edd1e-79af-4ceb-85a7-507a8f6de614-kube-api-access-nb6p6\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:05.096496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.095698    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504edd1e-79af-4ceb-85a7-507a8f6de614-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:05.191776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.191734    4851 generic.go:356] "Generic (PLEG): container finished" podID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerID="21551b73ce361fa492dbeb429008926bf3e85ee228dbc18887210c7c3e466f51" exitCode=60
Nov 02 01:45:05.192250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.191823    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" event={"ID":"93ffddad-f58a-4f2a-8eca-0a95c5ac330d","Type":"ContainerDied","Data":"21551b73ce361fa492dbeb429008926bf3e85ee228dbc18887210c7c3e466f51"}
Nov 02 01:45:05.192250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.191890    4851 scope.go:122] "RemoveContainer" containerID="79e9ef96ece48a1f1fae061a6c0048efdac5b75423e2e0ecf441570e3ef3a5fc"
Nov 02 01:45:05.192250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.192156    4851 scope.go:122] "RemoveContainer" containerID="21551b73ce361fa492dbeb429008926bf3e85ee228dbc18887210c7c3e466f51"
Nov 02 01:45:05.192594 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:05.192571    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk_odh-ai-gateway-infra(93ffddad-f58a-4f2a-8eca-0a95c5ac330d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d"
Nov 02 01:45:05.194038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.194018    4851 scope.go:122] "RemoveContainer" containerID="4dcbf341aa6a601ed8ada1eec9b8fa9c73101f99519a3a06d63dc1b8587a6b90"
Nov 02 01:45:05.194398 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:05.194379    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng_odh-ai-gateway-infra(d965b026-5f19-49bd-9cd4-9c8e2c16ecc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0"
Nov 02 01:45:05.195813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.195788    4851 generic.go:356] "Generic (PLEG): container finished" podID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerID="5ce22cd7667a418b174e5b44573dffa3774e02529579e6452e945c048895aa5b" exitCode=60
Nov 02 01:45:05.195927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.195870    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" event={"ID":"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134","Type":"ContainerDied","Data":"5ce22cd7667a418b174e5b44573dffa3774e02529579e6452e945c048895aa5b"}
Nov 02 01:45:05.196183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.196167    4851 scope.go:122] "RemoveContainer" containerID="5ce22cd7667a418b174e5b44573dffa3774e02529579e6452e945c048895aa5b"
Nov 02 01:45:05.196469 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:05.196439    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt_odh-ai-gateway-infra(caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134"
Nov 02 01:45:05.196601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.196474    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.196991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.196966    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mlnw\" (UniqueName: \"kubernetes.io/projected/a781d41c-a182-493a-bb96-b346b2ba2f01-kube-api-access-4mlnw\") pod \"maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.197187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.197161    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.197492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.197474    4851 generic.go:356] "Generic (PLEG): container finished" podID="504edd1e-79af-4ceb-85a7-507a8f6de614" containerID="c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24" exitCode=0
Nov 02 01:45:05.197667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.197522    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"
Nov 02 01:45:05.197751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.197518    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" event={"ID":"504edd1e-79af-4ceb-85a7-507a8f6de614","Type":"ContainerDied","Data":"c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24"}
Nov 02 01:45:05.197814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.197788    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb" event={"ID":"504edd1e-79af-4ceb-85a7-507a8f6de614","Type":"ContainerDied","Data":"c41d8fa415f42ddcffc54ada60473b2347db8c2dac96c2fa137f6aab319c02e8"}
Nov 02 01:45:05.199639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.199616    4851 generic.go:356] "Generic (PLEG): container finished" podID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerID="71f294437be909815f8df3f3d42c27fa19fef6f3fda8b0b1d4c1afd7bff3ebb7" exitCode=60
Nov 02 01:45:05.199747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.199722    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" event={"ID":"0d49aac2-82f1-48cf-91c9-07298ba7b4c3","Type":"ContainerDied","Data":"71f294437be909815f8df3f3d42c27fa19fef6f3fda8b0b1d4c1afd7bff3ebb7"}
Nov 02 01:45:05.199882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.199860    4851 scope.go:122] "RemoveContainer" containerID="71f294437be909815f8df3f3d42c27fa19fef6f3fda8b0b1d4c1afd7bff3ebb7"
Nov 02 01:45:05.200084 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:05.200059    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m_odh-ai-gateway-infra(0d49aac2-82f1-48cf-91c9-07298ba7b4c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3"
Nov 02 01:45:05.202161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.202138    4851 generic.go:356] "Generic (PLEG): container finished" podID="119a2d02-b88d-4052-b81c-a43a441ff466" containerID="92cd14779073bdae9470632c847137fac0017d7266ccd1e72841308bb813d476" exitCode=60
Nov 02 01:45:05.202268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.202216    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" event={"ID":"119a2d02-b88d-4052-b81c-a43a441ff466","Type":"ContainerDied","Data":"92cd14779073bdae9470632c847137fac0017d7266ccd1e72841308bb813d476"}
Nov 02 01:45:05.202432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.202414    4851 scope.go:122] "RemoveContainer" containerID="92cd14779073bdae9470632c847137fac0017d7266ccd1e72841308bb813d476"
Nov 02 01:45:05.202648 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:05.202627    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn_odh-ai-gateway-infra(119a2d02-b88d-4052-b81c-a43a441ff466)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" podUID="119a2d02-b88d-4052-b81c-a43a441ff466"
Nov 02 01:45:05.209464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.204708    4851 generic.go:356] "Generic (PLEG): container finished" podID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerID="03b633b463a7a934ebe9c3b133e8089cc79d1104f76e7de0bdf7d19b597cbcae" exitCode=60
Nov 02 01:45:05.209464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.204829    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" event={"ID":"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab","Type":"ContainerDied","Data":"03b633b463a7a934ebe9c3b133e8089cc79d1104f76e7de0bdf7d19b597cbcae"}
Nov 02 01:45:05.209464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.205131    4851 scope.go:122] "RemoveContainer" containerID="03b633b463a7a934ebe9c3b133e8089cc79d1104f76e7de0bdf7d19b597cbcae"
Nov 02 01:45:05.209464 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:05.205423    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893065-bst7j_odh-ai-gateway-infra(bbbc6c42-07a0-412d-8a9d-36ce3c0686ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab"
Nov 02 01:45:05.211666 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.211649    4851 scope.go:122] "RemoveContainer" containerID="8b68bedf4f2a413beb81cf8bd2b98702f56f748f6d2fb9eaba8fb61be6f7d8b2"
Nov 02 01:45:05.212965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.212938    4851 generic.go:356] "Generic (PLEG): container finished" podID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerID="5f565e637e74f68f962c8bc9683f1fa0d04e2a1682dc26a2689d8a0e518712d9" exitCode=60
Nov 02 01:45:05.213372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.213063    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh" event={"ID":"26f9bef1-c9ff-4de5-82e1-43eb7a1709da","Type":"ContainerDied","Data":"5f565e637e74f68f962c8bc9683f1fa0d04e2a1682dc26a2689d8a0e518712d9"}
Nov 02 01:45:05.213372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.213234    4851 scope.go:122] "RemoveContainer" containerID="5f565e637e74f68f962c8bc9683f1fa0d04e2a1682dc26a2689d8a0e518712d9"
Nov 02 01:45:05.213570 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:05.213551    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh_odh-ai-gateway-infra(26f9bef1-c9ff-4de5-82e1-43eb7a1709da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da"
Nov 02 01:45:05.215370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.215347    4851 generic.go:356] "Generic (PLEG): container finished" podID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerID="4653bae9622dce45e14810e9d1928adb35ecc94fe668964b9b9b3dad0a3479c0" exitCode=60
Nov 02 01:45:05.215462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.215398    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x" event={"ID":"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45","Type":"ContainerDied","Data":"4653bae9622dce45e14810e9d1928adb35ecc94fe668964b9b9b3dad0a3479c0"}
Nov 02 01:45:05.215621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.215608    4851 scope.go:122] "RemoveContainer" containerID="4653bae9622dce45e14810e9d1928adb35ecc94fe668964b9b9b3dad0a3479c0"
Nov 02 01:45:05.215829 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:05.215814    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x_odh-ai-gateway-infra(9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45"
Nov 02 01:45:05.227494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.227476    4851 scope.go:122] "RemoveContainer" containerID="c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24"
Nov 02 01:45:05.231724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.231700    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"]
Nov 02 01:45:05.241691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.241639    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-xhnrb"]
Nov 02 01:45:05.243104 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.243088    4851 scope.go:122] "RemoveContainer" containerID="c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24"
Nov 02 01:45:05.243526 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:05.243459    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24\": container with ID starting with c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24 not found: ID does not exist" containerID="c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24"
Nov 02 01:45:05.243526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.243482    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24"} err="failed to get container status \"c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24\": rpc error: code = NotFound desc = could not find container \"c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24\": container with ID starting with c618dd4eec0f852d4972674fc4001447543b6b252a9d79321d2842bd6f919e24 not found: ID does not exist"
Nov 02 01:45:05.243526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.243498    4851 scope.go:122] "RemoveContainer" containerID="6a6fa11caf40f5f5ba45fd742b5d9ff76bbbbef313239c8eadf24008c1e8a482"
Nov 02 01:45:05.261944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.261919    4851 scope.go:122] "RemoveContainer" containerID="aa6da8765d787061103dc21913c4751cdcb2baded1412c26d66124955f974abd"
Nov 02 01:45:05.281182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.281160    4851 scope.go:122] "RemoveContainer" containerID="640a9cf67206b5f94ad8558d0cc0154ea814d2f52cda1bc0741f4175ea43d146"
Nov 02 01:45:05.297085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.297065    4851 scope.go:122] "RemoveContainer" containerID="a6bd784f3c92a9c9192ea2a23460367bbc59cf9266beb208994e184930e96e1f"
Nov 02 01:45:05.298805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.298778    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mlnw\" (UniqueName: \"kubernetes.io/projected/a781d41c-a182-493a-bb96-b346b2ba2f01-kube-api-access-4mlnw\") pod \"maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.299144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.299114    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.299766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.299746    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.300343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.300276    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.301793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.301774    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.311981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.311957    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mlnw\" (UniqueName: \"kubernetes.io/projected/a781d41c-a182-493a-bb96-b346b2ba2f01-kube-api-access-4mlnw\") pod \"maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.316536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.316510    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:05.316969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.316943    4851 scope.go:122] "RemoveContainer" containerID="866a0510a084ceae140d1c9a6f686ee7dd505bfed7fa24a90ef48ae978f95add"
Nov 02 01:45:05.758944 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:05.758914    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda781d41c_a182_493a_bb96_b346b2ba2f01.slice/crio-7273386785ac521097ef5f1ffde5f0aa12055580eae3644e137ba91ca09bc4a7 WatchSource:0}: Error finding container 7273386785ac521097ef5f1ffde5f0aa12055580eae3644e137ba91ca09bc4a7: Status 404 returned error can't find the container with id 7273386785ac521097ef5f1ffde5f0aa12055580eae3644e137ba91ca09bc4a7
Nov 02 01:45:05.760566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:05.760436    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"]
Nov 02 01:45:06.229952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.229922    4851 scope.go:122] "RemoveContainer" containerID="4653bae9622dce45e14810e9d1928adb35ecc94fe668964b9b9b3dad0a3479c0"
Nov 02 01:45:06.230426 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:06.230370    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x_odh-ai-gateway-infra(9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45"
Nov 02 01:45:06.231934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.231910    4851 scope.go:122] "RemoveContainer" containerID="21551b73ce361fa492dbeb429008926bf3e85ee228dbc18887210c7c3e466f51"
Nov 02 01:45:06.232272 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:06.232251    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk_odh-ai-gateway-infra(93ffddad-f58a-4f2a-8eca-0a95c5ac330d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d"
Nov 02 01:45:06.233232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.233203    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r" event={"ID":"a781d41c-a182-493a-bb96-b346b2ba2f01","Type":"ContainerStarted","Data":"7273386785ac521097ef5f1ffde5f0aa12055580eae3644e137ba91ca09bc4a7"}
Nov 02 01:45:06.235818 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.235798    4851 scope.go:122] "RemoveContainer" containerID="5ce22cd7667a418b174e5b44573dffa3774e02529579e6452e945c048895aa5b"
Nov 02 01:45:06.236143 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:06.236120    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt_odh-ai-gateway-infra(caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134"
Nov 02 01:45:06.238578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.238556    4851 scope.go:122] "RemoveContainer" containerID="71f294437be909815f8df3f3d42c27fa19fef6f3fda8b0b1d4c1afd7bff3ebb7"
Nov 02 01:45:06.238916 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:06.238893    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m_odh-ai-gateway-infra(0d49aac2-82f1-48cf-91c9-07298ba7b4c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3"
Nov 02 01:45:06.240483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.240393    4851 scope.go:122] "RemoveContainer" containerID="92cd14779073bdae9470632c847137fac0017d7266ccd1e72841308bb813d476"
Nov 02 01:45:06.240702 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:06.240681    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn_odh-ai-gateway-infra(119a2d02-b88d-4052-b81c-a43a441ff466)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" podUID="119a2d02-b88d-4052-b81c-a43a441ff466"
Nov 02 01:45:06.243212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.242896    4851 scope.go:122] "RemoveContainer" containerID="03b633b463a7a934ebe9c3b133e8089cc79d1104f76e7de0bdf7d19b597cbcae"
Nov 02 01:45:06.243212 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:06.243194    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893065-bst7j_odh-ai-gateway-infra(bbbc6c42-07a0-412d-8a9d-36ce3c0686ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab"
Nov 02 01:45:06.245172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.245150    4851 scope.go:122] "RemoveContainer" containerID="5f565e637e74f68f962c8bc9683f1fa0d04e2a1682dc26a2689d8a0e518712d9"
Nov 02 01:45:06.245465 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:06.245444    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh_odh-ai-gateway-infra(26f9bef1-c9ff-4de5-82e1-43eb7a1709da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da"
Nov 02 01:45:06.914711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.914677    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504edd1e-79af-4ceb-85a7-507a8f6de614" path="/var/lib/kubelet/pods/504edd1e-79af-4ceb-85a7-507a8f6de614/volumes"
Nov 02 01:45:06.924657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.924627    4851 patch_prober.go:29] interesting pod/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:45:06.924827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:06.924683    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:45:07.319363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:07.311481    4851 generic.go:356] "Generic (PLEG): container finished" podID="a781d41c-a182-493a-bb96-b346b2ba2f01" containerID="7e1dfe401d59f064cd51e09d2b12f33e3f089e92f72d9b8cf4b56c63a15c2c78" exitCode=0
Nov 02 01:45:07.319363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:07.311722    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r" event={"ID":"a781d41c-a182-493a-bb96-b346b2ba2f01","Type":"ContainerDied","Data":"7e1dfe401d59f064cd51e09d2b12f33e3f089e92f72d9b8cf4b56c63a15c2c78"}
Nov 02 01:45:08.710441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:08.710411    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:08.787921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:08.787879    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-openshift-ca\") pod \"a781d41c-a182-493a-bb96-b346b2ba2f01\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") "
Nov 02 01:45:08.788122 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:08.788083    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a781d41c-a182-493a-bb96-b346b2ba2f01-kube-api-access-4mlnw\" (UniqueName: \"kubernetes.io/projected/a781d41c-a182-493a-bb96-b346b2ba2f01-kube-api-access-4mlnw\") pod \"a781d41c-a182-493a-bb96-b346b2ba2f01\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") "
Nov 02 01:45:08.788272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:08.788200    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-xks-ca\") pod \"a781d41c-a182-493a-bb96-b346b2ba2f01\" (UID: \"a781d41c-a182-493a-bb96-b346b2ba2f01\") "
Nov 02 01:45:08.789315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:08.789271    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-openshift-ca" pod "a781d41c-a182-493a-bb96-b346b2ba2f01" (UID: "a781d41c-a182-493a-bb96-b346b2ba2f01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:08.791983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:08.791823    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-xks-ca" pod "a781d41c-a182-493a-bb96-b346b2ba2f01" (UID: "a781d41c-a182-493a-bb96-b346b2ba2f01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:45:08.792568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:08.792541    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a781d41c-a182-493a-bb96-b346b2ba2f01-kube-api-access-4mlnw" pod "a781d41c-a182-493a-bb96-b346b2ba2f01" (UID: "a781d41c-a182-493a-bb96-b346b2ba2f01"). InnerVolumeSpecName "kube-api-access-4mlnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:08.898314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:08.897546    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mlnw\" (UniqueName: \"kubernetes.io/projected/a781d41c-a182-493a-bb96-b346b2ba2f01-kube-api-access-4mlnw\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:08.898314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:08.897581    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:08.898314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:08.897597    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a781d41c-a182-493a-bb96-b346b2ba2f01-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:09.239317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.232882    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"]
Nov 02 01:45:09.239317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.235251    4851 cpu_manager.go:414] "containerMap: removing container" podUID="a781d41c-a182-493a-bb96-b346b2ba2f01" containerName="revoke-keys"
Nov 02 01:45:09.239317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.235277    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a781d41c-a182-493a-bb96-b346b2ba2f01" containerName="revoke-keys"
Nov 02 01:45:09.239317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.235528    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="a781d41c-a182-493a-bb96-b346b2ba2f01" containerName="revoke-keys"
Nov 02 01:45:09.258315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.246771    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.275950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.272903    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"]
Nov 02 01:45:09.372461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.372378    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r" event={"ID":"a781d41c-a182-493a-bb96-b346b2ba2f01","Type":"ContainerDied","Data":"7273386785ac521097ef5f1ffde5f0aa12055580eae3644e137ba91ca09bc4a7"}
Nov 02 01:45:09.372461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.372416    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"
Nov 02 01:45:09.372461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.372439    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7273386785ac521097ef5f1ffde5f0aa12055580eae3644e137ba91ca09bc4a7"
Nov 02 01:45:09.410277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.410244    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009155c0-v8sr6\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.410277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.410278    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009155c0-v8sr6\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.410581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.410429    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fxb6\" (UniqueName: \"kubernetes.io/projected/56b544bc-e8ac-429d-aa90-452a2e294d16-kube-api-access-7fxb6\") pod \"maas-api-revoke-sub-009155c0-v8sr6\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.524712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.520669    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009155c0-v8sr6\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.524712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.520718    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009155c0-v8sr6\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.524712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.520785    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fxb6\" (UniqueName: \"kubernetes.io/projected/56b544bc-e8ac-429d-aa90-452a2e294d16-kube-api-access-7fxb6\") pod \"maas-api-revoke-sub-009155c0-v8sr6\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.524712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.522139    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009155c0-v8sr6\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.535764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.535710    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009155c0-v8sr6\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.546312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.536073    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fxb6\" (UniqueName: \"kubernetes.io/projected/56b544bc-e8ac-429d-aa90-452a2e294d16-kube-api-access-7fxb6\") pod \"maas-api-revoke-sub-009155c0-v8sr6\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.574513 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.574480    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:09.891927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:09.891892    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"]
Nov 02 01:45:09.895610 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:09.895578    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b544bc_e8ac_429d_aa90_452a2e294d16.slice/crio-de5f2f257da169aa19d9e65f7b083cb64b066f99abefb37be3b90c4c3ea50878 WatchSource:0}: Error finding container de5f2f257da169aa19d9e65f7b083cb64b066f99abefb37be3b90c4c3ea50878: Status 404 returned error can't find the container with id de5f2f257da169aa19d9e65f7b083cb64b066f99abefb37be3b90c4c3ea50878
Nov 02 01:45:10.011886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.011857    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:45:10.029742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.029710    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-kube-api-access-4kw2s\" (UniqueName: \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-kube-api-access-4kw2s\") pod \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") "
Nov 02 01:45:10.029905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.029754    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-certs\") pod \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") "
Nov 02 01:45:10.029905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.029779    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-token\" (UniqueName: \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-token\") pod \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") "
Nov 02 01:45:10.029905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.029854    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-podinfo\") pod \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") "
Nov 02 01:45:10.029905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.029892    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-credential-socket\") pod \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") "
Nov 02 01:45:10.030219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.030127    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-data\") pod \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") "
Nov 02 01:45:10.030219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.030162    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istiod-ca-cert\") pod \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") "
Nov 02 01:45:10.030219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.030175    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-certs" pod "6bda2b76-d865-47d7-9c84-e01ffbd776f4" (UID: "6bda2b76-d865-47d7-9c84-e01ffbd776f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:10.030428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.030223    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-socket\") pod \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") "
Nov 02 01:45:10.030428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.030367    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-envoy\") pod \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\" (UID: \"6bda2b76-d865-47d7-9c84-e01ffbd776f4\") "
Nov 02 01:45:10.030428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.030388    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-credential-socket" pod "6bda2b76-d865-47d7-9c84-e01ffbd776f4" (UID: "6bda2b76-d865-47d7-9c84-e01ffbd776f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:10.030607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.030449    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-data" pod "6bda2b76-d865-47d7-9c84-e01ffbd776f4" (UID: "6bda2b76-d865-47d7-9c84-e01ffbd776f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:10.030668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.030602    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istiod-ca-cert" pod "6bda2b76-d865-47d7-9c84-e01ffbd776f4" (UID: "6bda2b76-d865-47d7-9c84-e01ffbd776f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:10.030668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.030621    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-socket" pod "6bda2b76-d865-47d7-9c84-e01ffbd776f4" (UID: "6bda2b76-d865-47d7-9c84-e01ffbd776f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:10.031048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.031028    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.031137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.031052    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.031137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.031065    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.031137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.031077    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.031137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.031090    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.068436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.060743    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-podinfo" pod "6bda2b76-d865-47d7-9c84-e01ffbd776f4" (UID: "6bda2b76-d865-47d7-9c84-e01ffbd776f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:45:10.068436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.066914    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-kube-api-access-4kw2s" pod "6bda2b76-d865-47d7-9c84-e01ffbd776f4" (UID: "6bda2b76-d865-47d7-9c84-e01ffbd776f4"). InnerVolumeSpecName "kube-api-access-4kw2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:10.069119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.068900    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-envoy" pod "6bda2b76-d865-47d7-9c84-e01ffbd776f4" (UID: "6bda2b76-d865-47d7-9c84-e01ffbd776f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:10.080921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.080873    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-token" pod "6bda2b76-d865-47d7-9c84-e01ffbd776f4" (UID: "6bda2b76-d865-47d7-9c84-e01ffbd776f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:10.197407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.160595    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.197407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.160635    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kw2s\" (UniqueName: \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-kube-api-access-4kw2s\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.197407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.160654    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.197407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.160668    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bda2b76-d865-47d7-9c84-e01ffbd776f4-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.386109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.386078    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6" event={"ID":"56b544bc-e8ac-429d-aa90-452a2e294d16","Type":"ContainerStarted","Data":"de5f2f257da169aa19d9e65f7b083cb64b066f99abefb37be3b90c4c3ea50878"}
Nov 02 01:45:10.388193 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.388164    4851 generic.go:356] "Generic (PLEG): container finished" podID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerID="8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121" exitCode=0
Nov 02 01:45:10.388340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.388271    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"
Nov 02 01:45:10.388340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.388328    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" event={"ID":"6bda2b76-d865-47d7-9c84-e01ffbd776f4","Type":"ContainerDied","Data":"8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121"}
Nov 02 01:45:10.388436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.388402    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475" event={"ID":"6bda2b76-d865-47d7-9c84-e01ffbd776f4","Type":"ContainerDied","Data":"47f62ca65caf0094573d5bb734fc8432badc406ebbaef26f36fd216a8d8001a5"}
Nov 02 01:45:10.388436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.388427    4851 scope.go:122] "RemoveContainer" containerID="8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121"
Nov 02 01:45:10.463599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.463508    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"]
Nov 02 01:45:10.471407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.471038    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-41215f-openshift-default-7cb4d865d5-rf475"]
Nov 02 01:45:10.482277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.482160    4851 scope.go:122] "RemoveContainer" containerID="eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80"
Nov 02 01:45:10.497443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.497413    4851 scope.go:122] "RemoveContainer" containerID="8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121"
Nov 02 01:45:10.497916 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:10.497886    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121\": container with ID starting with 8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121 not found: ID does not exist" containerID="8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121"
Nov 02 01:45:10.498031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.497932    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121"} err="failed to get container status \"8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121\": rpc error: code = NotFound desc = could not find container \"8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121\": container with ID starting with 8d1e8fe4ca96e40cab753e94c69b1f887416b8f60397e9c580ac8905ddd03121 not found: ID does not exist"
Nov 02 01:45:10.498031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.497961    4851 scope.go:122] "RemoveContainer" containerID="eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80"
Nov 02 01:45:10.498354 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:10.498269    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80\": container with ID starting with eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80 not found: ID does not exist" containerID="eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80"
Nov 02 01:45:10.498677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.498355    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80"} err="failed to get container status \"eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80\": rpc error: code = NotFound desc = could not find container \"eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80\": container with ID starting with eb897925d1c7336f4f16a53ce50a3186022aab5e9ba9b4c8ea26ea9fbb281f80 not found: ID does not exist"
Nov 02 01:45:10.707629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.707608    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:45:10.775534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.775428    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578eb675-6e20-4ed1-a546-a0bb01f3f899-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578eb675-6e20-4ed1-a546-a0bb01f3f899-plugins-config-volume\") pod \"578eb675-6e20-4ed1-a546-a0bb01f3f899\" (UID: \"578eb675-6e20-4ed1-a546-a0bb01f3f899\") "
Nov 02 01:45:10.775812 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.775791    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578eb675-6e20-4ed1-a546-a0bb01f3f899-kube-api-access-d4twh\" (UniqueName: \"kubernetes.io/projected/578eb675-6e20-4ed1-a546-a0bb01f3f899-kube-api-access-d4twh\") pod \"578eb675-6e20-4ed1-a546-a0bb01f3f899\" (UID: \"578eb675-6e20-4ed1-a546-a0bb01f3f899\") "
Nov 02 01:45:10.775902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.775813    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578eb675-6e20-4ed1-a546-a0bb01f3f899-plugins-config-volume" pod "578eb675-6e20-4ed1-a546-a0bb01f3f899" (UID: "578eb675-6e20-4ed1-a546-a0bb01f3f899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:10.776364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.776345    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578eb675-6e20-4ed1-a546-a0bb01f3f899-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.778275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.778245    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578eb675-6e20-4ed1-a546-a0bb01f3f899-kube-api-access-d4twh" pod "578eb675-6e20-4ed1-a546-a0bb01f3f899" (UID: "578eb675-6e20-4ed1-a546-a0bb01f3f899"). InnerVolumeSpecName "kube-api-access-d4twh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:10.819560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.819534    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:45:10.901319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.879751    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec4c675-7706-4edf-808a-a542b0986037-kube-api-access-zsfrf\" (UniqueName: \"kubernetes.io/projected/2ec4c675-7706-4edf-808a-a542b0986037-kube-api-access-zsfrf\") pod \"2ec4c675-7706-4edf-808a-a542b0986037\" (UID: \"2ec4c675-7706-4edf-808a-a542b0986037\") "
Nov 02 01:45:10.901319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.879900    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec4c675-7706-4edf-808a-a542b0986037-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec4c675-7706-4edf-808a-a542b0986037-plugins-config-volume\") pod \"2ec4c675-7706-4edf-808a-a542b0986037\" (UID: \"2ec4c675-7706-4edf-808a-a542b0986037\") "
Nov 02 01:45:10.901319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.880688    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4twh\" (UniqueName: \"kubernetes.io/projected/578eb675-6e20-4ed1-a546-a0bb01f3f899-kube-api-access-d4twh\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.902310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.902223    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec4c675-7706-4edf-808a-a542b0986037-plugins-config-volume" pod "2ec4c675-7706-4edf-808a-a542b0986037" (UID: "2ec4c675-7706-4edf-808a-a542b0986037"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:10.976675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.960706    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec4c675-7706-4edf-808a-a542b0986037-kube-api-access-zsfrf" pod "2ec4c675-7706-4edf-808a-a542b0986037" (UID: "2ec4c675-7706-4edf-808a-a542b0986037"). InnerVolumeSpecName "kube-api-access-zsfrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:10.995264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.993963    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsfrf\" (UniqueName: \"kubernetes.io/projected/2ec4c675-7706-4edf-808a-a542b0986037-kube-api-access-zsfrf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.995264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.993998    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec4c675-7706-4edf-808a-a542b0986037-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:10.999517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:10.999481    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" path="/var/lib/kubelet/pods/6bda2b76-d865-47d7-9c84-e01ffbd776f4/volumes"
Nov 02 01:45:11.399032 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.398942    4851 generic.go:356] "Generic (PLEG): container finished" podID="56b544bc-e8ac-429d-aa90-452a2e294d16" containerID="4d5413c76706ea2197b5a4d911547d2cf90683e6ae8d26158802f2ecdcac3cc2" exitCode=0
Nov 02 01:45:11.399213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.399037    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6" event={"ID":"56b544bc-e8ac-429d-aa90-452a2e294d16","Type":"ContainerDied","Data":"4d5413c76706ea2197b5a4d911547d2cf90683e6ae8d26158802f2ecdcac3cc2"}
Nov 02 01:45:11.400529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.400505    4851 generic.go:356] "Generic (PLEG): container finished" podID="578eb675-6e20-4ed1-a546-a0bb01f3f899" containerID="44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f" exitCode=0
Nov 02 01:45:11.400644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.400551    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds" event={"ID":"578eb675-6e20-4ed1-a546-a0bb01f3f899","Type":"ContainerDied","Data":"44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f"}
Nov 02 01:45:11.400644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.400598    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds" event={"ID":"578eb675-6e20-4ed1-a546-a0bb01f3f899","Type":"ContainerDied","Data":"06b290d0c009374f967ad4a283dab02ed9dfbe4d55bd1ebbd26a06df105e6c36"}
Nov 02 01:45:11.400644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.400628    4851 scope.go:122] "RemoveContainer" containerID="44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f"
Nov 02 01:45:11.400644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.400554    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"
Nov 02 01:45:11.403544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.403519    4851 generic.go:356] "Generic (PLEG): container finished" podID="2ec4c675-7706-4edf-808a-a542b0986037" containerID="b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001" exitCode=0
Nov 02 01:45:11.403629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.403562    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5" event={"ID":"2ec4c675-7706-4edf-808a-a542b0986037","Type":"ContainerDied","Data":"b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001"}
Nov 02 01:45:11.403629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.403578    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"
Nov 02 01:45:11.403629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.403612    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5" event={"ID":"2ec4c675-7706-4edf-808a-a542b0986037","Type":"ContainerDied","Data":"b68a5e85fa7c34d5a1820a28cb89a7175db709f13029be4861d45f5cee3b903f"}
Nov 02 01:45:11.413261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.413239    4851 scope.go:122] "RemoveContainer" containerID="44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f"
Nov 02 01:45:11.413575 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:11.413554    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f\": container with ID starting with 44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f not found: ID does not exist" containerID="44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f"
Nov 02 01:45:11.413645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.413582    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f"} err="failed to get container status \"44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f\": rpc error: code = NotFound desc = could not find container \"44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f\": container with ID starting with 44a438b6d7a42e665597d05bd57ccb22301db8f966cc29d62bc9f13d9c94f40f not found: ID does not exist"
Nov 02 01:45:11.413645 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.413600    4851 scope.go:122] "RemoveContainer" containerID="b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001"
Nov 02 01:45:11.424332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.424313    4851 scope.go:122] "RemoveContainer" containerID="b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001"
Nov 02 01:45:11.424638 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:11.424617    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001\": container with ID starting with b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001 not found: ID does not exist" containerID="b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001"
Nov 02 01:45:11.424679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.424645    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001"} err="failed to get container status \"b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001\": rpc error: code = NotFound desc = could not find container \"b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001\": container with ID starting with b8e085e1ed3644dad78ab468d53dd76d84d44473d1dff0f58460b7fa993f4001 not found: ID does not exist"
Nov 02 01:45:11.503602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.503566    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"]
Nov 02 01:45:11.507886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.507854    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-41215f-6794bd58b7-qq7x5"]
Nov 02 01:45:11.520363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.520335    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"]
Nov 02 01:45:11.524596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:11.524568    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-41215f-c9775c9d7-7wbds"]
Nov 02 01:45:12.722011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.721977    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:12.825374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.825319    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b544bc-e8ac-429d-aa90-452a2e294d16-kube-api-access-7fxb6\" (UniqueName: \"kubernetes.io/projected/56b544bc-e8ac-429d-aa90-452a2e294d16-kube-api-access-7fxb6\") pod \"56b544bc-e8ac-429d-aa90-452a2e294d16\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") "
Nov 02 01:45:12.825602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.825537    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-openshift-ca\") pod \"56b544bc-e8ac-429d-aa90-452a2e294d16\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") "
Nov 02 01:45:12.825681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.825614    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-xks-ca\") pod \"56b544bc-e8ac-429d-aa90-452a2e294d16\" (UID: \"56b544bc-e8ac-429d-aa90-452a2e294d16\") "
Nov 02 01:45:12.825969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.825944    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-openshift-ca" pod "56b544bc-e8ac-429d-aa90-452a2e294d16" (UID: "56b544bc-e8ac-429d-aa90-452a2e294d16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:12.826560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.826530    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:12.827955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.827927    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-xks-ca" pod "56b544bc-e8ac-429d-aa90-452a2e294d16" (UID: "56b544bc-e8ac-429d-aa90-452a2e294d16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:45:12.828152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.828134    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b544bc-e8ac-429d-aa90-452a2e294d16-kube-api-access-7fxb6" pod "56b544bc-e8ac-429d-aa90-452a2e294d16" (UID: "56b544bc-e8ac-429d-aa90-452a2e294d16"). InnerVolumeSpecName "kube-api-access-7fxb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:12.870362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.870320    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec4c675-7706-4edf-808a-a542b0986037" path="/var/lib/kubelet/pods/2ec4c675-7706-4edf-808a-a542b0986037/volumes"
Nov 02 01:45:12.871052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.871032    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578eb675-6e20-4ed1-a546-a0bb01f3f899" path="/var/lib/kubelet/pods/578eb675-6e20-4ed1-a546-a0bb01f3f899/volumes"
Nov 02 01:45:12.928210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.928172    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b544bc-e8ac-429d-aa90-452a2e294d16-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:12.928210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:12.928203    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fxb6\" (UniqueName: \"kubernetes.io/projected/56b544bc-e8ac-429d-aa90-452a2e294d16-kube-api-access-7fxb6\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:13.428205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:13.428111    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"
Nov 02 01:45:13.428205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:13.428140    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6" event={"ID":"56b544bc-e8ac-429d-aa90-452a2e294d16","Type":"ContainerDied","Data":"de5f2f257da169aa19d9e65f7b083cb64b066f99abefb37be3b90c4c3ea50878"}
Nov 02 01:45:13.428205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:13.428191    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de5f2f257da169aa19d9e65f7b083cb64b066f99abefb37be3b90c4c3ea50878"
Nov 02 01:45:14.846912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:14.846885    4851 scope.go:122] "RemoveContainer" containerID="4653bae9622dce45e14810e9d1928adb35ecc94fe668964b9b9b3dad0a3479c0"
Nov 02 01:45:15.070963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:15.070868    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"]
Nov 02 01:45:15.087768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:15.087736    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"]
Nov 02 01:45:15.840049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:15.839685    4851 scope.go:122] "RemoveContainer" containerID="71f294437be909815f8df3f3d42c27fa19fef6f3fda8b0b1d4c1afd7bff3ebb7"
Nov 02 01:45:16.472745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:16.472710    4851 generic.go:356] "Generic (PLEG): container finished" podID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerID="555ae39ed3d6abe8808f29e4e31ec7382ceeacf32df70b631a8fd19bdb56eef2" exitCode=60
Nov 02 01:45:16.473276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:16.472759    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x" event={"ID":"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45","Type":"ContainerDied","Data":"555ae39ed3d6abe8808f29e4e31ec7382ceeacf32df70b631a8fd19bdb56eef2"}
Nov 02 01:45:16.473276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:16.472826    4851 scope.go:122] "RemoveContainer" containerID="4653bae9622dce45e14810e9d1928adb35ecc94fe668964b9b9b3dad0a3479c0"
Nov 02 01:45:16.473276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:16.473198    4851 scope.go:122] "RemoveContainer" containerID="555ae39ed3d6abe8808f29e4e31ec7382ceeacf32df70b631a8fd19bdb56eef2"
Nov 02 01:45:16.473657 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:16.473629    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x_odh-ai-gateway-infra(9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45"
Nov 02 01:45:16.475493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:16.475468    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" event={"ID":"0d49aac2-82f1-48cf-91c9-07298ba7b4c3","Type":"ContainerStarted","Data":"1f67c001ea305bb98410caca076aea1984afe2a6b8363d23475197cd0af07a04"}
Nov 02 01:45:16.839958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:16.839925    4851 scope.go:122] "RemoveContainer" containerID="5ce22cd7667a418b174e5b44573dffa3774e02529579e6452e945c048895aa5b"
Nov 02 01:45:16.840603 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:16.840578    4851 scope.go:122] "RemoveContainer" containerID="4dcbf341aa6a601ed8ada1eec9b8fa9c73101f99519a3a06d63dc1b8587a6b90"
Nov 02 01:45:16.841334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:16.841283    4851 scope.go:122] "RemoveContainer" containerID="92cd14779073bdae9470632c847137fac0017d7266ccd1e72841308bb813d476"
Nov 02 01:45:17.487464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.487429    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng" event={"ID":"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0","Type":"ContainerStarted","Data":"0e2e195a60b0cb9c4d5be12701efbaa7323ac31a82f0dd34d19100ecc8a582ae"}
Nov 02 01:45:17.489698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.489667    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" event={"ID":"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134","Type":"ContainerStarted","Data":"3dadc54f2409c8712820edeff0c8a72eb2afaba35b901e711ca07e9ec21f5b11"}
Nov 02 01:45:17.492010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.491980    4851 generic.go:356] "Generic (PLEG): container finished" podID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerID="1f67c001ea305bb98410caca076aea1984afe2a6b8363d23475197cd0af07a04" exitCode=60
Nov 02 01:45:17.492137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.492039    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" event={"ID":"0d49aac2-82f1-48cf-91c9-07298ba7b4c3","Type":"ContainerDied","Data":"1f67c001ea305bb98410caca076aea1984afe2a6b8363d23475197cd0af07a04"}
Nov 02 01:45:17.492137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.492084    4851 scope.go:122] "RemoveContainer" containerID="71f294437be909815f8df3f3d42c27fa19fef6f3fda8b0b1d4c1afd7bff3ebb7"
Nov 02 01:45:17.492414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.492396    4851 scope.go:122] "RemoveContainer" containerID="1f67c001ea305bb98410caca076aea1984afe2a6b8363d23475197cd0af07a04"
Nov 02 01:45:17.492717 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:17.492691    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m_odh-ai-gateway-infra(0d49aac2-82f1-48cf-91c9-07298ba7b4c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3"
Nov 02 01:45:17.494336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.494310    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" event={"ID":"119a2d02-b88d-4052-b81c-a43a441ff466","Type":"ContainerStarted","Data":"2a9956e5f538adbaa76516a53090e08320f4a53024ce9d48e27d5e8934fe955c"}
Nov 02 01:45:17.504261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.504237    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"]
Nov 02 01:45:17.508556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.508513    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng" podStartSLOduration=17.508501633 podStartE2EDuration="17.508501633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:17.506333309 +0000 UTC m=+1822.811266137" watchObservedRunningTime="2026-11-02 01:45:17.508501633 +0000 UTC m=+1822.813434458"
Nov 02 01:45:17.617353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.592068    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"]
Nov 02 01:45:17.666798 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:17.666720    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:45:17.816482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.816452    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"
Nov 02 01:45:17.841892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.841191    4851 scope.go:122] "RemoveContainer" containerID="5f565e637e74f68f962c8bc9683f1fa0d04e2a1682dc26a2689d8a0e518712d9"
Nov 02 01:45:17.841892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.841493    4851 scope.go:122] "RemoveContainer" containerID="21551b73ce361fa492dbeb429008926bf3e85ee228dbc18887210c7c3e466f51"
Nov 02 01:45:17.946249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.946225    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": read tcp 10.132.0.2:42718->10.132.0.234:15021: read: connection reset by peer" start-of-body=
Nov 02 01:45:17.946426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.946279    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": read tcp 10.132.0.2:42718->10.132.0.234:15021: read: connection reset by peer"
Nov 02 01:45:17.967172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.967145    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-kube-api-access-kjx9d\" (UniqueName: \"kubernetes.io/projected/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-kube-api-access-kjx9d\") pod \"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45\" (UID: \"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45\") "
Nov 02 01:45:17.967432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.967412    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-maas-api-service-ca\") pod \"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45\" (UID: \"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45\") "
Nov 02 01:45:17.967759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.967738    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-maas-api-service-ca" pod "9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" (UID: "9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:17.968327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.968267    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:17.969381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:17.969285    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-kube-api-access-kjx9d" pod "9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" (UID: "9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45"). InnerVolumeSpecName "kube-api-access-kjx9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:18.070445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.070379    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjx9d\" (UniqueName: \"kubernetes.io/projected/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45-kube-api-access-kjx9d\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:18.505260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.505225    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh" event={"ID":"26f9bef1-c9ff-4de5-82e1-43eb7a1709da","Type":"ContainerStarted","Data":"e3d28049d902a24383652b1de6f927038e9898761ec5b3d41e1b35b4e1401abc"}
Nov 02 01:45:18.506747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.506726    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"
Nov 02 01:45:18.506910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.506728    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x" event={"ID":"9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45","Type":"ContainerDied","Data":"8b5faf4e62a78cadb117592c8299781fb586315d81ea3896754e8bc70dc707c3"}
Nov 02 01:45:18.506910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.506850    4851 scope.go:122] "RemoveContainer" containerID="555ae39ed3d6abe8808f29e4e31ec7382ceeacf32df70b631a8fd19bdb56eef2"
Nov 02 01:45:18.508648 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.508618    4851 generic.go:356] "Generic (PLEG): container finished" podID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerID="0511219424eac1bed532c7b2fe2bc593937ff78743c479f5b01dfe7d6a1a728a" exitCode=0
Nov 02 01:45:18.508788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.508682    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" event={"ID":"fa8656c6-8347-438b-b2ee-b2ae03c69590","Type":"ContainerDied","Data":"0511219424eac1bed532c7b2fe2bc593937ff78743c479f5b01dfe7d6a1a728a"}
Nov 02 01:45:18.509162 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.509114    4851 scope.go:122] "RemoveContainer" containerID="0511219424eac1bed532c7b2fe2bc593937ff78743c479f5b01dfe7d6a1a728a"
Nov 02 01:45:18.511073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.511022    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:18.511197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.511107    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:18.511197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.511159    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:18.512151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.512129    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" event={"ID":"93ffddad-f58a-4f2a-8eca-0a95c5ac330d","Type":"ContainerStarted","Data":"8d86d5f48d46e88e475c8f4e3cbec2585d341b613a9540bd137de92fe0489bd8"}
Nov 02 01:45:18.515944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.515918    4851 generic.go:356] "Generic (PLEG): container finished" podID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerID="0e2e195a60b0cb9c4d5be12701efbaa7323ac31a82f0dd34d19100ecc8a582ae" exitCode=60
Nov 02 01:45:18.516064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.515992    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng" event={"ID":"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0","Type":"ContainerDied","Data":"0e2e195a60b0cb9c4d5be12701efbaa7323ac31a82f0dd34d19100ecc8a582ae"}
Nov 02 01:45:18.516411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.516390    4851 scope.go:122] "RemoveContainer" containerID="0e2e195a60b0cb9c4d5be12701efbaa7323ac31a82f0dd34d19100ecc8a582ae"
Nov 02 01:45:18.516698 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:18.516671    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng_odh-ai-gateway-infra(d965b026-5f19-49bd-9cd4-9c8e2c16ecc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0"
Nov 02 01:45:18.519048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.518354    4851 generic.go:356] "Generic (PLEG): container finished" podID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerID="3dadc54f2409c8712820edeff0c8a72eb2afaba35b901e711ca07e9ec21f5b11" exitCode=60
Nov 02 01:45:18.519048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.518485    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" event={"ID":"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134","Type":"ContainerDied","Data":"3dadc54f2409c8712820edeff0c8a72eb2afaba35b901e711ca07e9ec21f5b11"}
Nov 02 01:45:18.519048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.518799    4851 scope.go:122] "RemoveContainer" containerID="3dadc54f2409c8712820edeff0c8a72eb2afaba35b901e711ca07e9ec21f5b11"
Nov 02 01:45:18.519048 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:18.519013    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt_odh-ai-gateway-infra(caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134"
Nov 02 01:45:18.521127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.521108    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"]
Nov 02 01:45:18.525378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.524459    4851 scope.go:122] "RemoveContainer" containerID="4dcbf341aa6a601ed8ada1eec9b8fa9c73101f99519a3a06d63dc1b8587a6b90"
Nov 02 01:45:18.526467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.526443    4851 generic.go:356] "Generic (PLEG): container finished" podID="119a2d02-b88d-4052-b81c-a43a441ff466" containerID="2a9956e5f538adbaa76516a53090e08320f4a53024ce9d48e27d5e8934fe955c" exitCode=60
Nov 02 01:45:18.526606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.526592    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" event={"ID":"119a2d02-b88d-4052-b81c-a43a441ff466","Type":"ContainerDied","Data":"2a9956e5f538adbaa76516a53090e08320f4a53024ce9d48e27d5e8934fe955c"}
Nov 02 01:45:18.526963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.526947    4851 scope.go:122] "RemoveContainer" containerID="2a9956e5f538adbaa76516a53090e08320f4a53024ce9d48e27d5e8934fe955c"
Nov 02 01:45:18.527265 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:18.527245    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn_odh-ai-gateway-infra(119a2d02-b88d-4052-b81c-a43a441ff466)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" podUID="119a2d02-b88d-4052-b81c-a43a441ff466"
Nov 02 01:45:18.548853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.548823    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"]
Nov 02 01:45:18.550508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.550487    4851 scope.go:122] "RemoveContainer" containerID="5ce22cd7667a418b174e5b44573dffa3774e02529579e6452e945c048895aa5b"
Nov 02 01:45:18.575740 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.575712    4851 scope.go:122] "RemoveContainer" containerID="92cd14779073bdae9470632c847137fac0017d7266ccd1e72841308bb813d476"
Nov 02 01:45:18.588370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.586638    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"]
Nov 02 01:45:18.707223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.707137    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"]
Nov 02 01:45:18.720194 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.720111    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f689bc-29893065-m5d6x"]
Nov 02 01:45:18.803864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.803838    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"
Nov 02 01:45:18.840208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.839774    4851 scope.go:122] "RemoveContainer" containerID="03b633b463a7a934ebe9c3b133e8089cc79d1104f76e7de0bdf7d19b597cbcae"
Nov 02 01:45:18.853817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.853784    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" path="/var/lib/kubelet/pods/9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45/volumes"
Nov 02 01:45:18.896021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.895991    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-kube-api-access-qcnsg\" (UniqueName: \"kubernetes.io/projected/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-kube-api-access-qcnsg\") pod \"0d49aac2-82f1-48cf-91c9-07298ba7b4c3\" (UID: \"0d49aac2-82f1-48cf-91c9-07298ba7b4c3\") "
Nov 02 01:45:18.896220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.896121    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-maas-api-service-ca\") pod \"0d49aac2-82f1-48cf-91c9-07298ba7b4c3\" (UID: \"0d49aac2-82f1-48cf-91c9-07298ba7b4c3\") "
Nov 02 01:45:18.896514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.896491    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-maas-api-service-ca" pod "0d49aac2-82f1-48cf-91c9-07298ba7b4c3" (UID: "0d49aac2-82f1-48cf-91c9-07298ba7b4c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:18.896793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.896777    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:18.898748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.898723    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-kube-api-access-qcnsg" pod "0d49aac2-82f1-48cf-91c9-07298ba7b4c3" (UID: "0d49aac2-82f1-48cf-91c9-07298ba7b4c3"). InnerVolumeSpecName "kube-api-access-qcnsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:18.999662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:18.999581    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcnsg\" (UniqueName: \"kubernetes.io/projected/0d49aac2-82f1-48cf-91c9-07298ba7b4c3-kube-api-access-qcnsg\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:19.560413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.560374    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"]
Nov 02 01:45:19.568228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.568198    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" event={"ID":"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab","Type":"ContainerStarted","Data":"b7c3409fbf68885ce792c6fa015be46538e9067eff717afbabe86c2f20e9f06c"}
Nov 02 01:45:19.570764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.570727    4851 generic.go:356] "Generic (PLEG): container finished" podID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerID="e3d28049d902a24383652b1de6f927038e9898761ec5b3d41e1b35b4e1401abc" exitCode=60
Nov 02 01:45:19.570904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.570828    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh" event={"ID":"26f9bef1-c9ff-4de5-82e1-43eb7a1709da","Type":"ContainerDied","Data":"e3d28049d902a24383652b1de6f927038e9898761ec5b3d41e1b35b4e1401abc"}
Nov 02 01:45:19.570904 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.570893    4851 scope.go:122] "RemoveContainer" containerID="5f565e637e74f68f962c8bc9683f1fa0d04e2a1682dc26a2689d8a0e518712d9"
Nov 02 01:45:19.575622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.575591    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" event={"ID":"fa8656c6-8347-438b-b2ee-b2ae03c69590","Type":"ContainerStarted","Data":"eece6565022ab35ef5494332e5255fcf64fe7473f113c96096f7660f7c9242f0"}
Nov 02 01:45:19.575737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.575711    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"]
Nov 02 01:45:19.579060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.579031    4851 generic.go:356] "Generic (PLEG): container finished" podID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerID="8d86d5f48d46e88e475c8f4e3cbec2585d341b613a9540bd137de92fe0489bd8" exitCode=60
Nov 02 01:45:19.579171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.579127    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" event={"ID":"93ffddad-f58a-4f2a-8eca-0a95c5ac330d","Type":"ContainerDied","Data":"8d86d5f48d46e88e475c8f4e3cbec2585d341b613a9540bd137de92fe0489bd8"}
Nov 02 01:45:19.590177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.590114    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m" event={"ID":"0d49aac2-82f1-48cf-91c9-07298ba7b4c3","Type":"ContainerDied","Data":"3598772b02b4a71547999a702b528ec0bc9d3be07d1abdb4bba24181565cb403"}
Nov 02 01:45:19.590303 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.590162    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"
Nov 02 01:45:19.594427 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.594399    4851 scope.go:122] "RemoveContainer" containerID="21551b73ce361fa492dbeb429008926bf3e85ee228dbc18887210c7c3e466f51"
Nov 02 01:45:19.631612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.631581    4851 scope.go:122] "RemoveContainer" containerID="1f67c001ea305bb98410caca076aea1984afe2a6b8363d23475197cd0af07a04"
Nov 02 01:45:19.830464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.830436    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"]
Nov 02 01:45:19.867317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.864777    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:45:19.867317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.864934    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 01:45:19.867317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.864979    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 01:45:19.872343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.872267    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-12e7cb-29893065-p5w2m"]
Nov 02 01:45:19.920734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:19.920001    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:20.076481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.076348    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx" podUID="90d90625-4204-4fec-8fb1-1237f358bd6a" containerName="payload-processing" containerID="cri-o://a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec" gracePeriod=25
Nov 02 01:45:20.112340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.095404    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z" podUID="0f7739d3-8fb8-453e-8af5-345d8bd842e4" containerName="payload-pre-processing" containerID="cri-o://42a03eccb342b409f039e155501c3bdcbb7662883e12e09bd6a1d3281fda6773" gracePeriod=25
Nov 02 01:45:20.202528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.139035    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119a2d02-b88d-4052-b81c-a43a441ff466-kube-api-access-rfshq\" (UniqueName: \"kubernetes.io/projected/119a2d02-b88d-4052-b81c-a43a441ff466-kube-api-access-rfshq\") pod \"119a2d02-b88d-4052-b81c-a43a441ff466\" (UID: \"119a2d02-b88d-4052-b81c-a43a441ff466\") "
Nov 02 01:45:20.202528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.139138    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119a2d02-b88d-4052-b81c-a43a441ff466-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/119a2d02-b88d-4052-b81c-a43a441ff466-maas-api-service-ca\") pod \"119a2d02-b88d-4052-b81c-a43a441ff466\" (UID: \"119a2d02-b88d-4052-b81c-a43a441ff466\") "
Nov 02 01:45:20.204902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.204833    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119a2d02-b88d-4052-b81c-a43a441ff466-maas-api-service-ca" pod "119a2d02-b88d-4052-b81c-a43a441ff466" (UID: "119a2d02-b88d-4052-b81c-a43a441ff466"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:20.286558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.258450    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/119a2d02-b88d-4052-b81c-a43a441ff466-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.313429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.291931    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"
Nov 02 01:45:20.313429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.295489    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119a2d02-b88d-4052-b81c-a43a441ff466-kube-api-access-rfshq" pod "119a2d02-b88d-4052-b81c-a43a441ff466" (UID: "119a2d02-b88d-4052-b81c-a43a441ff466"). InnerVolumeSpecName "kube-api-access-rfshq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:20.378566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.370662    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfshq\" (UniqueName: \"kubernetes.io/projected/119a2d02-b88d-4052-b81c-a43a441ff466-kube-api-access-rfshq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.399533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.399502    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"
Nov 02 01:45:20.507913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.472881    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-maas-api-service-ca\") pod \"93ffddad-f58a-4f2a-8eca-0a95c5ac330d\" (UID: \"93ffddad-f58a-4f2a-8eca-0a95c5ac330d\") "
Nov 02 01:45:20.507913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.472960    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-kube-api-access-d4t6x\" (UniqueName: \"kubernetes.io/projected/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-kube-api-access-d4t6x\") pod \"93ffddad-f58a-4f2a-8eca-0a95c5ac330d\" (UID: \"93ffddad-f58a-4f2a-8eca-0a95c5ac330d\") "
Nov 02 01:45:20.507913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.473049    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-kube-api-access-rwrfg\" (UniqueName: \"kubernetes.io/projected/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-kube-api-access-rwrfg\") pod \"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134\" (UID: \"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134\") "
Nov 02 01:45:20.507913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.473212    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-maas-api-service-ca\") pod \"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134\" (UID: \"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134\") "
Nov 02 01:45:20.507913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.507508    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-maas-api-service-ca" pod "caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" (UID: "caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:20.566734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.516157    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-maas-api-service-ca" pod "93ffddad-f58a-4f2a-8eca-0a95c5ac330d" (UID: "93ffddad-f58a-4f2a-8eca-0a95c5ac330d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:20.578597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.570444    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-kube-api-access-d4t6x" pod "93ffddad-f58a-4f2a-8eca-0a95c5ac330d" (UID: "93ffddad-f58a-4f2a-8eca-0a95c5ac330d"). InnerVolumeSpecName "kube-api-access-d4t6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:20.578597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.575314    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.578597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.575343    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4t6x\" (UniqueName: \"kubernetes.io/projected/93ffddad-f58a-4f2a-8eca-0a95c5ac330d-kube-api-access-d4t6x\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.578597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.575359    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.582358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.580067    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-kube-api-access-rwrfg" pod "caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" (UID: "caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134"). InnerVolumeSpecName "kube-api-access-rwrfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:20.596653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.596441    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"]
Nov 02 01:45:20.602778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.602681    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f689bc-ph77r"]
Nov 02 01:45:20.620352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.619660    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"]
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620433    4851 cpu_manager.go:414] "containerMap: removing container" podUID="2ec4c675-7706-4edf-808a-a542b0986037" containerName="payload-pre-processing"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620448    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec4c675-7706-4edf-808a-a542b0986037" containerName="payload-pre-processing"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620458    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620464    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620475    4851 cpu_manager.go:414] "containerMap: removing container" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620480    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620487    4851 cpu_manager.go:414] "containerMap: removing container" podUID="119a2d02-b88d-4052-b81c-a43a441ff466" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620491    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119a2d02-b88d-4052-b81c-a43a441ff466" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620506    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620511    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620518    4851 cpu_manager.go:414] "containerMap: removing container" podUID="578eb675-6e20-4ed1-a546-a0bb01f3f899" containerName="payload-processing"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620523    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578eb675-6e20-4ed1-a546-a0bb01f3f899" containerName="payload-processing"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620528    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620533    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620541    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620545    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620554    4851 cpu_manager.go:414] "containerMap: removing container" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerName="cleanup"
Nov 02 01:45:20.620547 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620559    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620566    4851 cpu_manager.go:414] "containerMap: removing container" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620571    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620591    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620595    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620603    4851 cpu_manager.go:414] "containerMap: removing container" podUID="56b544bc-e8ac-429d-aa90-452a2e294d16" containerName="revoke-keys"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620607    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b544bc-e8ac-429d-aa90-452a2e294d16" containerName="revoke-keys"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620616    4851 cpu_manager.go:414] "containerMap: removing container" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620622    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620628    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620632    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620641    4851 cpu_manager.go:414] "containerMap: removing container" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620645    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620658    4851 cpu_manager.go:414] "containerMap: removing container" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620662    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620679    4851 cpu_manager.go:414] "containerMap: removing container" podUID="119a2d02-b88d-4052-b81c-a43a441ff466" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620682    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119a2d02-b88d-4052-b81c-a43a441ff466" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620772    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620778    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b544bc-e8ac-429d-aa90-452a2e294d16" containerName="revoke-keys"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620784    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="119a2d02-b88d-4052-b81c-a43a441ff466" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620790    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620795    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620801    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620806    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bda2b76-d865-47d7-9c84-e01ffbd776f4" containerName="istio-proxy"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620814    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="119a2d02-b88d-4052-b81c-a43a441ff466" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620820    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec4c675-7706-4edf-808a-a542b0986037" containerName="payload-pre-processing"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620825    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620832    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620837    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="119a2d02-b88d-4052-b81c-a43a441ff466" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620844    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620848    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="578eb675-6e20-4ed1-a546-a0bb01f3f899" containerName="payload-processing"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620854    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620860    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620866    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerName="cleanup"
Nov 02 01:45:20.621644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.620870    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerName="cleanup"
Nov 02 01:45:20.622912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.622808    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"
Nov 02 01:45:20.624942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.624494    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.625652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.625628    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"
Nov 02 01:45:20.626773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.626734    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"]
Nov 02 01:45:20.628012 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.627991    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:45:20.631139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.631110    4851 generic.go:356] "Generic (PLEG): container finished" podID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerID="b7c3409fbf68885ce792c6fa015be46538e9067eff717afbabe86c2f20e9f06c" exitCode=60
Nov 02 01:45:20.631245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.631228    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" event={"ID":"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab","Type":"ContainerDied","Data":"b7c3409fbf68885ce792c6fa015be46538e9067eff717afbabe86c2f20e9f06c"}
Nov 02 01:45:20.631357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.631276    4851 scope.go:122] "RemoveContainer" containerID="03b633b463a7a934ebe9c3b133e8089cc79d1104f76e7de0bdf7d19b597cbcae"
Nov 02 01:45:20.648903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.631748    4851 scope.go:122] "RemoveContainer" containerID="b7c3409fbf68885ce792c6fa015be46538e9067eff717afbabe86c2f20e9f06c"
Nov 02 01:45:20.648903 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:20.632153    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893065-bst7j_odh-ai-gateway-infra(bbbc6c42-07a0-412d-8a9d-36ce3c0686ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab"
Nov 02 01:45:20.652578 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.652554    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"
Nov 02 01:45:20.652973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.652947    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh" event={"ID":"26f9bef1-c9ff-4de5-82e1-43eb7a1709da","Type":"ContainerDied","Data":"117d4e15252880aa56c9ebd025ed1893db7f89fca094aa92dc1fe610e7d7aabd"}
Nov 02 01:45:20.660049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.655756    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk" event={"ID":"93ffddad-f58a-4f2a-8eca-0a95c5ac330d","Type":"ContainerDied","Data":"0e31ec99de8ca9794d9d2c0c4c82b168fedd6b2214de8b0a5a48ac579723453e"}
Nov 02 01:45:20.660049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.655843    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.678416    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-maas-api-service-ca\") pod \"26f9bef1-c9ff-4de5-82e1-43eb7a1709da\" (UID: \"26f9bef1-c9ff-4de5-82e1-43eb7a1709da\") "
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.678671    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-kube-api-access-vbvwl\" (UniqueName: \"kubernetes.io/projected/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-kube-api-access-vbvwl\") pod \"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0\" (UID: \"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0\") "
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.678737    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-maas-api-service-ca\") pod \"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0\" (UID: \"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0\") "
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.678859    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-maas-api-service-ca" pod "26f9bef1-c9ff-4de5-82e1-43eb7a1709da" (UID: "26f9bef1-c9ff-4de5-82e1-43eb7a1709da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.678901    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-kube-api-access-k9x5t\" (UniqueName: \"kubernetes.io/projected/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-kube-api-access-k9x5t\") pod \"26f9bef1-c9ff-4de5-82e1-43eb7a1709da\" (UID: \"26f9bef1-c9ff-4de5-82e1-43eb7a1709da\") "
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.679185    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245r6\" (UniqueName: \"kubernetes.io/projected/90bc40f5-4ebc-4382-9182-40b1d463ba2e-kube-api-access-245r6\") pod \"maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.679324    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.679557    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.679683    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwrfg\" (UniqueName: \"kubernetes.io/projected/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134-kube-api-access-rwrfg\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.679700    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.680060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.679833    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-maas-api-service-ca" pod "d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" (UID: "d965b026-5f19-49bd-9cd4-9c8e2c16ecc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:20.681565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.681527    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-kube-api-access-vbvwl" pod "d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" (UID: "d965b026-5f19-49bd-9cd4-9c8e2c16ecc0"). InnerVolumeSpecName "kube-api-access-vbvwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:20.715388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.713752    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-kube-api-access-k9x5t" pod "26f9bef1-c9ff-4de5-82e1-43eb7a1709da" (UID: "26f9bef1-c9ff-4de5-82e1-43eb7a1709da"). InnerVolumeSpecName "kube-api-access-k9x5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:20.732138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.732090    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng" event={"ID":"d965b026-5f19-49bd-9cd4-9c8e2c16ecc0","Type":"ContainerDied","Data":"228399f5e9823fcecd711f9e4a22e3674ca864fcae7efc1a6379f333a09ecb8c"}
Nov 02 01:45:20.732495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.732475    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"
Nov 02 01:45:20.795381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.794609    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.795381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.794881    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.795381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.795025    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245r6\" (UniqueName: \"kubernetes.io/projected/90bc40f5-4ebc-4382-9182-40b1d463ba2e-kube-api-access-245r6\") pod \"maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.795381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.795142    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbvwl\" (UniqueName: \"kubernetes.io/projected/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-kube-api-access-vbvwl\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.795381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.795162    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.795381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.795176    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9x5t\" (UniqueName: \"kubernetes.io/projected/26f9bef1-c9ff-4de5-82e1-43eb7a1709da-kube-api-access-k9x5t\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:20.807006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.796819    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.808467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.808444    4851 scope.go:122] "RemoveContainer" containerID="e3d28049d902a24383652b1de6f927038e9898761ec5b3d41e1b35b4e1401abc"
Nov 02 01:45:20.809560 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.809535    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt" event={"ID":"caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134","Type":"ContainerDied","Data":"2d509ec4ac26fe25251bed2423551c8bd0b28e5310c475fb5b5d8c906812ddf6"}
Nov 02 01:45:20.809798 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.809778    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"
Nov 02 01:45:20.810591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.810274    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245r6\" (UniqueName: \"kubernetes.io/projected/90bc40f5-4ebc-4382-9182-40b1d463ba2e-kube-api-access-245r6\") pod \"maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.815642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.815624    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:20.816671 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.816641    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" event={"ID":"119a2d02-b88d-4052-b81c-a43a441ff466","Type":"ContainerDied","Data":"d2ac6d230e52d04c7d9a0e53cd5a7c1331b063b7fdb2c9fac86ff73cf7ee0b42"}
Nov 02 01:45:20.821387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.821360    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.824926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.824858    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"]
Nov 02 01:45:20.858433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.858388    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 01:45:20.858588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.858451    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 01:45:20.859480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.859457    4851 scope.go:122] "RemoveContainer" containerID="8d86d5f48d46e88e475c8f4e3cbec2585d341b613a9540bd137de92fe0489bd8"
Nov 02 01:45:20.880662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.880637    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d49aac2-82f1-48cf-91c9-07298ba7b4c3" path="/var/lib/kubelet/pods/0d49aac2-82f1-48cf-91c9-07298ba7b4c3/volumes"
Nov 02 01:45:20.882895 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.882877    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a781d41c-a182-493a-bb96-b346b2ba2f01" path="/var/lib/kubelet/pods/a781d41c-a182-493a-bb96-b346b2ba2f01/volumes"
Nov 02 01:45:20.883692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.883672    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"]
Nov 02 01:45:20.883777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.883702    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5ed42b-29893065-nq4jk"]
Nov 02 01:45:20.883777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.883719    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"]
Nov 02 01:45:20.883777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.883730    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-677312-29893065-nl8ng"]
Nov 02 01:45:20.918369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.918347    4851 scope.go:122] "RemoveContainer" containerID="0e2e195a60b0cb9c4d5be12701efbaa7323ac31a82f0dd34d19100ecc8a582ae"
Nov 02 01:45:20.927568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.927538    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"]
Nov 02 01:45:20.942726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.942704    4851 scope.go:122] "RemoveContainer" containerID="3dadc54f2409c8712820edeff0c8a72eb2afaba35b901e711ca07e9ec21f5b11"
Nov 02 01:45:20.964059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.964033    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:20.974218 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.974189    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-eaec8c-29893065-q9kwt"]
Nov 02 01:45:20.991691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.991662    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"]
Nov 02 01:45:20.993073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.993053    4851 scope.go:122] "RemoveContainer" containerID="2a9956e5f538adbaa76516a53090e08320f4a53024ce9d48e27d5e8934fe955c"
Nov 02 01:45:20.996615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:20.996591    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bf1e55-29893065-fqcmh"]
Nov 02 01:45:21.285349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:21.285320    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"]
Nov 02 01:45:21.287100 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:21.287074    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bc40f5_4ebc_4382_9182_40b1d463ba2e.slice/crio-1aabbc69cc57098606f461eee433dcabb8aa290d7a2da5b725acac053d47dac2 WatchSource:0}: Error finding container 1aabbc69cc57098606f461eee433dcabb8aa290d7a2da5b725acac053d47dac2: Status 404 returned error can't find the container with id 1aabbc69cc57098606f461eee433dcabb8aa290d7a2da5b725acac053d47dac2
Nov 02 01:45:21.849955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:21.849917    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp" event={"ID":"90bc40f5-4ebc-4382-9182-40b1d463ba2e","Type":"ContainerStarted","Data":"1aabbc69cc57098606f461eee433dcabb8aa290d7a2da5b725acac053d47dac2"}
Nov 02 01:45:21.871322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:21.869005    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 01:45:21.871322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:21.869068    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 01:45:22.208356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.208329    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"
Nov 02 01:45:22.310947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.310906    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-kube-api-access-kwkpw\" (UniqueName: \"kubernetes.io/projected/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-kube-api-access-kwkpw\") pod \"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab\" (UID: \"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab\") "
Nov 02 01:45:22.311239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.311219    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-maas-api-service-ca\") pod \"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab\" (UID: \"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab\") "
Nov 02 01:45:22.385054 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.385013    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-kube-api-access-kwkpw" pod "bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" (UID: "bbbc6c42-07a0-412d-8a9d-36ce3c0686ab"). InnerVolumeSpecName "kube-api-access-kwkpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:22.397933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.391797    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-maas-api-service-ca" pod "bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" (UID: "bbbc6c42-07a0-412d-8a9d-36ce3c0686ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:22.417438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.414669    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwkpw\" (UniqueName: \"kubernetes.io/projected/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-kube-api-access-kwkpw\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:22.518853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.518771    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:22.862378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.860421    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 01:45:22.862378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.860489    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 01:45:22.916550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.915189    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx" podUID="90d90625-4204-4fec-8fb1-1237f358bd6a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:45:22.917244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.917105    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z" podUID="0f7739d3-8fb8-453e-8af5-345d8bd842e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:45:22.933130 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.933100    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"
Nov 02 01:45:22.969317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.968318    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" path="/var/lib/kubelet/pods/26f9bef1-c9ff-4de5-82e1-43eb7a1709da/volumes"
Nov 02 01:45:22.969317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.968447    4851 generic.go:356] "Generic (PLEG): container finished" podID="90bc40f5-4ebc-4382-9182-40b1d463ba2e" containerID="0664d0d4e62bbe375396360b4c51b584674e5325f1a694a2460a200ec64ff0fd" exitCode=0
Nov 02 01:45:22.969317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.968964    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" path="/var/lib/kubelet/pods/93ffddad-f58a-4f2a-8eca-0a95c5ac330d/volumes"
Nov 02 01:45:22.972482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.969511    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" path="/var/lib/kubelet/pods/caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134/volumes"
Nov 02 01:45:22.972482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.971579    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" path="/var/lib/kubelet/pods/d965b026-5f19-49bd-9cd4-9c8e2c16ecc0/volumes"
Nov 02 01:45:22.972482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.972254    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j" event={"ID":"bbbc6c42-07a0-412d-8a9d-36ce3c0686ab","Type":"ContainerDied","Data":"c462459e29edf7f3cae53a89ec83b98e470a4f475b56eaffc7113b7874a285c2"}
Nov 02 01:45:22.972482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.972324    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp" event={"ID":"90bc40f5-4ebc-4382-9182-40b1d463ba2e","Type":"ContainerDied","Data":"0664d0d4e62bbe375396360b4c51b584674e5325f1a694a2460a200ec64ff0fd"}
Nov 02 01:45:22.972482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:22.972359    4851 scope.go:122] "RemoveContainer" containerID="b7c3409fbf68885ce792c6fa015be46538e9067eff717afbabe86c2f20e9f06c"
Nov 02 01:45:23.017628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:23.017601    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"]
Nov 02 01:45:23.021732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:23.021707    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-bst7j"]
Nov 02 01:45:23.051996 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:23.051960    4851 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbc6c42_07a0_412d_8a9d_36ce3c0686ab.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbc6c42_07a0_412d_8a9d_36ce3c0686ab.slice/crio-c462459e29edf7f3cae53a89ec83b98e470a4f475b56eaffc7113b7874a285c2\": RecentStats: unable to find data in memory cache]"
Nov 02 01:45:23.855712 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:23.855675    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 01:45:23.855988 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:23.855961    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 01:45:24.186889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.186861    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:24.273899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.273860    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bc40f5-4ebc-4382-9182-40b1d463ba2e-kube-api-access-245r6\" (UniqueName: \"kubernetes.io/projected/90bc40f5-4ebc-4382-9182-40b1d463ba2e-kube-api-access-245r6\") pod \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") "
Nov 02 01:45:24.274110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.273966    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-xks-ca\") pod \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") "
Nov 02 01:45:24.274177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.274162    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-openshift-ca\") pod \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\" (UID: \"90bc40f5-4ebc-4382-9182-40b1d463ba2e\") "
Nov 02 01:45:24.274610 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.274578    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-openshift-ca" pod "90bc40f5-4ebc-4382-9182-40b1d463ba2e" (UID: "90bc40f5-4ebc-4382-9182-40b1d463ba2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:24.274878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.274839    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:24.276241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.276218    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-xks-ca" pod "90bc40f5-4ebc-4382-9182-40b1d463ba2e" (UID: "90bc40f5-4ebc-4382-9182-40b1d463ba2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:45:24.276365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.276321    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bc40f5-4ebc-4382-9182-40b1d463ba2e-kube-api-access-245r6" pod "90bc40f5-4ebc-4382-9182-40b1d463ba2e" (UID: "90bc40f5-4ebc-4382-9182-40b1d463ba2e"). InnerVolumeSpecName "kube-api-access-245r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:24.376558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.376526    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-245r6\" (UniqueName: \"kubernetes.io/projected/90bc40f5-4ebc-4382-9182-40b1d463ba2e-kube-api-access-245r6\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:24.376558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.376562    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bc40f5-4ebc-4382-9182-40b1d463ba2e-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:24.858237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.857801    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 01:45:24.858237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.857859    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 01:45:24.875841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.875808    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" path="/var/lib/kubelet/pods/bbbc6c42-07a0-412d-8a9d-36ce3c0686ab/volumes"
Nov 02 01:45:24.989033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.988999    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"
Nov 02 01:45:24.989221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.989047    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp" event={"ID":"90bc40f5-4ebc-4382-9182-40b1d463ba2e","Type":"ContainerDied","Data":"1aabbc69cc57098606f461eee433dcabb8aa290d7a2da5b725acac053d47dac2"}
Nov 02 01:45:24.989221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:24.989119    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aabbc69cc57098606f461eee433dcabb8aa290d7a2da5b725acac053d47dac2"
Nov 02 01:45:25.857737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:25.857705    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:45:25.858205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:25.858188    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:45:25.858890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:25.858873    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:45:28.194942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.194908    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"]
Nov 02 01:45:28.195435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.195118    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" containerID="cri-o://3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda" gracePeriod=30
Nov 02 01:45:28.203946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.203916    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"]
Nov 02 01:45:28.204151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.204115    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy" containerID="cri-o://616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5" gracePeriod=30
Nov 02 01:45:28.740688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.740653    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"]
Nov 02 01:45:28.741470 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741451    4851 cpu_manager.go:414] "containerMap: removing container" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerName="cleanup"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741472    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerName="cleanup"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741493    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerName="cleanup"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741498    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerName="cleanup"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741505    4851 cpu_manager.go:414] "containerMap: removing container" podUID="90bc40f5-4ebc-4382-9182-40b1d463ba2e" containerName="revoke-keys"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741510    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bc40f5-4ebc-4382-9182-40b1d463ba2e" containerName="revoke-keys"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741518    4851 cpu_manager.go:414] "containerMap: removing container" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerName="cleanup"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741523    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerName="cleanup"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741544    4851 cpu_manager.go:414] "containerMap: removing container" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerName="cleanup"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741550    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerName="cleanup"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741562    4851 cpu_manager.go:414] "containerMap: removing container" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerName="cleanup"
Nov 02 01:45:28.741570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741569    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741579    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741584    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741590    4851 cpu_manager.go:414] "containerMap: removing container" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741595    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caeb8dd9-4826-4bbd-b7c9-aa0fcadb4134" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741601    4851 cpu_manager.go:414] "containerMap: removing container" podUID="119a2d02-b88d-4052-b81c-a43a441ff466" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741604    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119a2d02-b88d-4052-b81c-a43a441ff466" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741610    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741614    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741631    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741638    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741652    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741656    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741799    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741812    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741823    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9587a258-f1ba-4ce4-bba5-1c0e4ef8cd45" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741829    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ffddad-f58a-4f2a-8eca-0a95c5ac330d" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741837    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741842    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bc40f5-4ebc-4382-9182-40b1d463ba2e" containerName="revoke-keys"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741848    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741855    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741861    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f9bef1-c9ff-4de5-82e1-43eb7a1709da" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741868    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerName="cleanup"
Nov 02 01:45:28.741919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.741878    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerName="cleanup"
Nov 02 01:45:28.747039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.747014    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:28.753556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.753529    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"]
Nov 02 01:45:28.767922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.767887    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hdd\" (UniqueName: \"kubernetes.io/projected/32517f68-24d9-4b95-99bf-2cbcd7cfb599-kube-api-access-72hdd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:28.768072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.768041    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:28.768153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.768131    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:28.846493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.846462    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:45:28.846659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.846559    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:45:28.868990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.868957    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hdd\" (UniqueName: \"kubernetes.io/projected/32517f68-24d9-4b95-99bf-2cbcd7cfb599-kube-api-access-72hdd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:28.869151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.869022    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:28.869151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.869072    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:28.869694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.869660    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:28.871848 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.871821    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:28.881264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:28.881241    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hdd\" (UniqueName: \"kubernetes.io/projected/32517f68-24d9-4b95-99bf-2cbcd7cfb599-kube-api-access-72hdd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:29.068952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:29.068876    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:29.355223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:29.355192    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"]
Nov 02 01:45:29.355672 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:29.355649    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32517f68_24d9_4b95_99bf_2cbcd7cfb599.slice/crio-d3ce68bb2179b3698acfba57cd14ff27b48fd922a7a235e4aed031cae1b2c481 WatchSource:0}: Error finding container d3ce68bb2179b3698acfba57cd14ff27b48fd922a7a235e4aed031cae1b2c481: Status 404 returned error can't find the container with id d3ce68bb2179b3698acfba57cd14ff27b48fd922a7a235e4aed031cae1b2c481
Nov 02 01:45:30.050760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:30.050720    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv" event={"ID":"32517f68-24d9-4b95-99bf-2cbcd7cfb599","Type":"ContainerStarted","Data":"0786562154e5e9ad1d263c4313b12d1725ccfcd07f4c346f419b59f75bdc0fc9"}
Nov 02 01:45:30.050948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:30.050779    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv" event={"ID":"32517f68-24d9-4b95-99bf-2cbcd7cfb599","Type":"ContainerStarted","Data":"d3ce68bb2179b3698acfba57cd14ff27b48fd922a7a235e4aed031cae1b2c481"}
Nov 02 01:45:30.072964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:30.072909    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv" podStartSLOduration=2.0728960929999998 podStartE2EDuration="2.072896093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:30.068816006 +0000 UTC m=+1835.373748833" watchObservedRunningTime="2026-11-02 01:45:30.072896093 +0000 UTC m=+1835.377828919"
Nov 02 01:45:30.713283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:30.713243    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"]
Nov 02 01:45:30.730882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:30.730848    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"]
Nov 02 01:45:31.121929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:31.121789    4851 generic.go:356] "Generic (PLEG): container finished" podID="32517f68-24d9-4b95-99bf-2cbcd7cfb599" containerID="0786562154e5e9ad1d263c4313b12d1725ccfcd07f4c346f419b59f75bdc0fc9" exitCode=0
Nov 02 01:45:31.121929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:31.121866    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv" event={"ID":"32517f68-24d9-4b95-99bf-2cbcd7cfb599","Type":"ContainerDied","Data":"0786562154e5e9ad1d263c4313b12d1725ccfcd07f4c346f419b59f75bdc0fc9"}
Nov 02 01:45:31.207451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:31.207400    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:45:31.207659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:31.207491    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:45:32.295635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.295607    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:32.437372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.432576    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32517f68-24d9-4b95-99bf-2cbcd7cfb599-kube-api-access-72hdd\" (UniqueName: \"kubernetes.io/projected/32517f68-24d9-4b95-99bf-2cbcd7cfb599-kube-api-access-72hdd\") pod \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") "
Nov 02 01:45:32.437372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.432930    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-openshift-ca\") pod \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") "
Nov 02 01:45:32.437372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.433007    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-xks-ca\") pod \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\" (UID: \"32517f68-24d9-4b95-99bf-2cbcd7cfb599\") "
Nov 02 01:45:32.437372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.434707    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-openshift-ca" pod "32517f68-24d9-4b95-99bf-2cbcd7cfb599" (UID: "32517f68-24d9-4b95-99bf-2cbcd7cfb599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:32.437372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.436899    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-xks-ca" pod "32517f68-24d9-4b95-99bf-2cbcd7cfb599" (UID: "32517f68-24d9-4b95-99bf-2cbcd7cfb599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:45:32.448427 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.442580    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32517f68-24d9-4b95-99bf-2cbcd7cfb599-kube-api-access-72hdd" pod "32517f68-24d9-4b95-99bf-2cbcd7cfb599" (UID: "32517f68-24d9-4b95-99bf-2cbcd7cfb599"). InnerVolumeSpecName "kube-api-access-72hdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:32.534655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.534622    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:32.534655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.534652    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32517f68-24d9-4b95-99bf-2cbcd7cfb599-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:32.534876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.534669    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72hdd\" (UniqueName: \"kubernetes.io/projected/32517f68-24d9-4b95-99bf-2cbcd7cfb599-kube-api-access-72hdd\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:32.886403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.886364    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx" podUID="90d90625-4204-4fec-8fb1-1237f358bd6a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:45:32.886600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:32.886377    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z" podUID="0f7739d3-8fb8-453e-8af5-345d8bd842e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:45:33.144565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.144485    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"
Nov 02 01:45:33.144745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.144484    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv" event={"ID":"32517f68-24d9-4b95-99bf-2cbcd7cfb599","Type":"ContainerDied","Data":"d3ce68bb2179b3698acfba57cd14ff27b48fd922a7a235e4aed031cae1b2c481"}
Nov 02 01:45:33.144745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.144616    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3ce68bb2179b3698acfba57cd14ff27b48fd922a7a235e4aed031cae1b2c481"
Nov 02 01:45:33.456626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.456590    4851 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 02 01:45:33.457129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.456659    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 02 01:45:33.510954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.510930    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:45:33.658443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.658399    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-podinfo\") pod \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") "
Nov 02 01:45:33.658861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.658697    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-credential-socket\") pod \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") "
Nov 02 01:45:33.658861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.658756    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istiod-ca-cert\") pod \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") "
Nov 02 01:45:33.658861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.658799    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-token\" (UniqueName: \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-token\") pod \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") "
Nov 02 01:45:33.658861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.658844    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-certs\") pod \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") "
Nov 02 01:45:33.659251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.658884    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-envoy\") pod \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") "
Nov 02 01:45:33.659251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.659041    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-socket\") pod \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") "
Nov 02 01:45:33.659251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.659074    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-kube-api-access-nqvd5\" (UniqueName: \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-kube-api-access-nqvd5\") pod \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") "
Nov 02 01:45:33.659251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.659162    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-data\") pod \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\" (UID: \"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e\") "
Nov 02 01:45:33.660751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.660062    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-data" pod "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" (UID: "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:33.661766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.661738    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-certs" pod "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" (UID: "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:33.661766 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.661754    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-socket" pod "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" (UID: "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:33.661933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.661908    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-credential-socket" pod "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" (UID: "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:33.662160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.662138    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istiod-ca-cert" pod "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" (UID: "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:33.664416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.664387    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-podinfo" pod "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" (UID: "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:45:33.665328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.664772    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-token" pod "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" (UID: "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:33.665328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.664981    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-kube-api-access-nqvd5" pod "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" (UID: "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e"). InnerVolumeSpecName "kube-api-access-nqvd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:33.665328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.665006    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-envoy" pod "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" (UID: "5d430d5d-2307-41d0-a3fb-9576ce2b2c1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:33.761754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.761722    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:33.761754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.761755    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:33.761946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.761770    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqvd5\" (UniqueName: \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-kube-api-access-nqvd5\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:33.761946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.761785    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:33.761946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.761798    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:33.761946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.761810    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:33.761946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.761822    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:33.761946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.761835    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:33.761946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.761847    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:33.922918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:33.922894    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:45:34.065360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065251    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-envoy\") pod \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") "
Nov 02 01:45:34.065360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065357    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-podinfo\") pod \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") "
Nov 02 01:45:34.065607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065457    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-credential-socket\") pod \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") "
Nov 02 01:45:34.065607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065482    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istiod-ca-cert\") pod \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") "
Nov 02 01:45:34.065607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065519    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-data\") pod \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") "
Nov 02 01:45:34.065607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065553    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-kube-api-access-g7p5q\" (UniqueName: \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-kube-api-access-g7p5q\") pod \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") "
Nov 02 01:45:34.065607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065603    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-certs\") pod \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") "
Nov 02 01:45:34.065908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065740    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-socket\") pod \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") "
Nov 02 01:45:34.065908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065768    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-token\" (UniqueName: \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-token\") pod \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\" (UID: \"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead\") "
Nov 02 01:45:34.065908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065869    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-credential-socket" pod "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" (UID: "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:34.066068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.065973    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istiod-ca-cert" pod "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" (UID: "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:34.066263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.066102    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-certs" pod "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" (UID: "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:34.066263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.066134    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-data" pod "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" (UID: "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:34.066440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.066329    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-socket" pod "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" (UID: "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:34.066573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.066524    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:34.066701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.066636    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:34.066760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.066722    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:34.066760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.066737    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:34.066874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.066775    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:34.067921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.067895    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-podinfo" pod "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" (UID: "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:45:34.068051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.067967    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-envoy" pod "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" (UID: "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:34.068118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.068088    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-kube-api-access-g7p5q" pod "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" (UID: "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead"). InnerVolumeSpecName "kube-api-access-g7p5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:34.068608 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.068588    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-token" pod "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" (UID: "016dfa35-82cc-4ab0-a117-b5f6a9bb2ead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:34.156304 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.156256    4851 generic.go:356] "Generic (PLEG): container finished" podID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerID="616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5" exitCode=0
Nov 02 01:45:34.156451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.156331    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"
Nov 02 01:45:34.156451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.156337    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" event={"ID":"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e","Type":"ContainerDied","Data":"616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5"}
Nov 02 01:45:34.156451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.156400    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd" event={"ID":"5d430d5d-2307-41d0-a3fb-9576ce2b2c1e","Type":"ContainerDied","Data":"3ed15a638d0a1d0390841d712bfb15095ad887c43e041370d4c454ad371a1e1c"}
Nov 02 01:45:34.156451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.156423    4851 scope.go:122] "RemoveContainer" containerID="616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5"
Nov 02 01:45:34.158500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.158479    4851 generic.go:356] "Generic (PLEG): container finished" podID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerID="3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda" exitCode=0
Nov 02 01:45:34.158616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.158523    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"
Nov 02 01:45:34.158616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.158540    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" event={"ID":"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead","Type":"ContainerDied","Data":"3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda"}
Nov 02 01:45:34.158616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.158575    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt" event={"ID":"016dfa35-82cc-4ab0-a117-b5f6a9bb2ead","Type":"ContainerDied","Data":"7d7539daaef63570cb9fdd2901325e5693db1f67215a6beb741f7fedd36c39de"}
Nov 02 01:45:34.168611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.168586    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7p5q\" (UniqueName: \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-kube-api-access-g7p5q\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:34.168743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.168615    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:34.168743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.168628    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:34.168743 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.168640    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:34.170640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.170615    4851 scope.go:122] "RemoveContainer" containerID="616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5"
Nov 02 01:45:34.170974 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:34.170953    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5\": container with ID starting with 616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5 not found: ID does not exist" containerID="616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5"
Nov 02 01:45:34.171038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.170980    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5"} err="failed to get container status \"616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5\": rpc error: code = NotFound desc = could not find container \"616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5\": container with ID starting with 616b51d69ed6591be89d8bed7154c93118f1fcb66aca325b4c8e6f7b543f82b5 not found: ID does not exist"
Nov 02 01:45:34.171038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.170996    4851 scope.go:122] "RemoveContainer" containerID="3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda"
Nov 02 01:45:34.182867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.182843    4851 scope.go:122] "RemoveContainer" containerID="d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df"
Nov 02 01:45:34.187956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.187926    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"]
Nov 02 01:45:34.192263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.192239    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-ccpkt"]
Nov 02 01:45:34.194800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.194783    4851 scope.go:122] "RemoveContainer" containerID="3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda"
Nov 02 01:45:34.195078 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:34.195059    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda\": container with ID starting with 3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda not found: ID does not exist" containerID="3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda"
Nov 02 01:45:34.195161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.195088    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda"} err="failed to get container status \"3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda\": rpc error: code = NotFound desc = could not find container \"3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda\": container with ID starting with 3151a9f647b9fbcd32a7052e6df43f8570b4cd2842ed44d696adc18eef564cda not found: ID does not exist"
Nov 02 01:45:34.195161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.195111    4851 scope.go:122] "RemoveContainer" containerID="d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df"
Nov 02 01:45:34.195370 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:34.195352    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df\": container with ID starting with d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df not found: ID does not exist" containerID="d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df"
Nov 02 01:45:34.195413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.195376    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df"} err="failed to get container status \"d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df\": rpc error: code = NotFound desc = could not find container \"d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df\": container with ID starting with d1094519189e5ec6e250bba3dd851cbedcec0143f0c06f870d733baafeeb65df not found: ID does not exist"
Nov 02 01:45:34.215171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.215129    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"]
Nov 02 01:45:34.219772 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.219746    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-hzbrd"]
Nov 02 01:45:34.848298 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.848257    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" path="/var/lib/kubelet/pods/016dfa35-82cc-4ab0-a117-b5f6a9bb2ead/volumes"
Nov 02 01:45:34.848815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:34.848799    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" path="/var/lib/kubelet/pods/5d430d5d-2307-41d0-a3fb-9576ce2b2c1e/volumes"
Nov 02 01:45:35.714429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:35.714389    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m" podUID="7d5a5b02-0b4a-4b54-a8e5-417b50f91a06" containerName="payload-processing" containerID="cri-o://40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0" gracePeriod=25
Nov 02 01:45:35.732072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:35.732018    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz" podUID="5eafdd80-7407-40d6-9579-cedb558b6411" containerName="payload-pre-processing" containerID="cri-o://679ca1dd80b9558dc896e5342c5463cf5ee3d884fe796199e92f43a1e5436601" gracePeriod=25
Nov 02 01:45:36.057691 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.057664    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:45:36.202320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.184237    4851 generic.go:356] "Generic (PLEG): container finished" podID="7d5a5b02-0b4a-4b54-a8e5-417b50f91a06" containerID="40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0" exitCode=0
Nov 02 01:45:36.202320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.184437    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m" event={"ID":"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06","Type":"ContainerDied","Data":"40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0"}
Nov 02 01:45:36.202320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.184505    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m" event={"ID":"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06","Type":"ContainerDied","Data":"1a29a457590a2de0e9d3f3a58283466809bed682dc11e53cc2162e4167eacba3"}
Nov 02 01:45:36.202320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.184533    4851 scope.go:122] "RemoveContainer" containerID="40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0"
Nov 02 01:45:36.202320 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.184651    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"
Nov 02 01:45:36.253339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.220796    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-kube-api-access-q74b5\" (UniqueName: \"kubernetes.io/projected/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-kube-api-access-q74b5\") pod \"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06\" (UID: \"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06\") "
Nov 02 01:45:36.253339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.251616    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-plugins-config-volume\") pod \"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06\" (UID: \"7d5a5b02-0b4a-4b54-a8e5-417b50f91a06\") "
Nov 02 01:45:36.253339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.252084    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-plugins-config-volume" pod "7d5a5b02-0b4a-4b54-a8e5-417b50f91a06" (UID: "7d5a5b02-0b4a-4b54-a8e5-417b50f91a06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:36.253339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.252984    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:36.265324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.261612    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-kube-api-access-q74b5" pod "7d5a5b02-0b4a-4b54-a8e5-417b50f91a06" (UID: "7d5a5b02-0b4a-4b54-a8e5-417b50f91a06"). InnerVolumeSpecName "kube-api-access-q74b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:36.358345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.356458    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q74b5\" (UniqueName: \"kubernetes.io/projected/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06-kube-api-access-q74b5\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:36.358345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.357325    4851 scope.go:122] "RemoveContainer" containerID="40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0"
Nov 02 01:45:36.376197 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:36.373452    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0\": container with ID starting with 40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0 not found: ID does not exist" containerID="40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0"
Nov 02 01:45:36.376197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.373507    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0"} err="failed to get container status \"40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0\": rpc error: code = NotFound desc = could not find container \"40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0\": container with ID starting with 40352f0f7ad0b593e1dc9b06bfceddaf8d414e70e44f7a11b9e2211668afaea0 not found: ID does not exist"
Nov 02 01:45:36.512768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.512693    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"]
Nov 02 01:45:36.521884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.521834    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf1e55-6664485d9-8vv2m"]
Nov 02 01:45:36.878321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:36.874882    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5a5b02-0b4a-4b54-a8e5-417b50f91a06" path="/var/lib/kubelet/pods/7d5a5b02-0b4a-4b54-a8e5-417b50f91a06/volumes"
Nov 02 01:45:38.970573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:38.970538    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"]
Nov 02 01:45:38.992254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:38.992217    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"]
Nov 02 01:45:40.252946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.252608    4851 generic.go:356] "Generic (PLEG): container finished" podID="0f7739d3-8fb8-453e-8af5-345d8bd842e4" containerID="42a03eccb342b409f039e155501c3bdcbb7662883e12e09bd6a1d3281fda6773" exitCode=0
Nov 02 01:45:40.252946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.252863    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z" event={"ID":"0f7739d3-8fb8-453e-8af5-345d8bd842e4","Type":"ContainerDied","Data":"42a03eccb342b409f039e155501c3bdcbb7662883e12e09bd6a1d3281fda6773"}
Nov 02 01:45:40.598216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.598175    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:45:40.619860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.619812    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7739d3-8fb8-453e-8af5-345d8bd842e4-kube-api-access-7nlk2\" (UniqueName: \"kubernetes.io/projected/0f7739d3-8fb8-453e-8af5-345d8bd842e4-kube-api-access-7nlk2\") pod \"0f7739d3-8fb8-453e-8af5-345d8bd842e4\" (UID: \"0f7739d3-8fb8-453e-8af5-345d8bd842e4\") "
Nov 02 01:45:40.620062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.620023    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f7739d3-8fb8-453e-8af5-345d8bd842e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7739d3-8fb8-453e-8af5-345d8bd842e4-plugins-config-volume\") pod \"0f7739d3-8fb8-453e-8af5-345d8bd842e4\" (UID: \"0f7739d3-8fb8-453e-8af5-345d8bd842e4\") "
Nov 02 01:45:40.620725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.620692    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7739d3-8fb8-453e-8af5-345d8bd842e4-plugins-config-volume" pod "0f7739d3-8fb8-453e-8af5-345d8bd842e4" (UID: "0f7739d3-8fb8-453e-8af5-345d8bd842e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:40.623141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.623110    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7739d3-8fb8-453e-8af5-345d8bd842e4-kube-api-access-7nlk2" pod "0f7739d3-8fb8-453e-8af5-345d8bd842e4" (UID: "0f7739d3-8fb8-453e-8af5-345d8bd842e4"). InnerVolumeSpecName "kube-api-access-7nlk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:40.816375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.812866    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nlk2\" (UniqueName: \"kubernetes.io/projected/0f7739d3-8fb8-453e-8af5-345d8bd842e4-kube-api-access-7nlk2\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:40.816649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.816624    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7739d3-8fb8-453e-8af5-345d8bd842e4-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:40.893316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.892623    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz" podUID="5eafdd80-7407-40d6-9579-cedb558b6411" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:45:40.915332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:40.915005    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:45:41.020585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.020556    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d90625-4204-4fec-8fb1-1237f358bd6a-kube-api-access-m2nd7\" (UniqueName: \"kubernetes.io/projected/90d90625-4204-4fec-8fb1-1237f358bd6a-kube-api-access-m2nd7\") pod \"90d90625-4204-4fec-8fb1-1237f358bd6a\" (UID: \"90d90625-4204-4fec-8fb1-1237f358bd6a\") "
Nov 02 01:45:41.020791 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.020647    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d90625-4204-4fec-8fb1-1237f358bd6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d90625-4204-4fec-8fb1-1237f358bd6a-plugins-config-volume\") pod \"90d90625-4204-4fec-8fb1-1237f358bd6a\" (UID: \"90d90625-4204-4fec-8fb1-1237f358bd6a\") "
Nov 02 01:45:41.021261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.021240    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d90625-4204-4fec-8fb1-1237f358bd6a-plugins-config-volume" pod "90d90625-4204-4fec-8fb1-1237f358bd6a" (UID: "90d90625-4204-4fec-8fb1-1237f358bd6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:41.024245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.024219    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d90625-4204-4fec-8fb1-1237f358bd6a-kube-api-access-m2nd7" pod "90d90625-4204-4fec-8fb1-1237f358bd6a" (UID: "90d90625-4204-4fec-8fb1-1237f358bd6a"). InnerVolumeSpecName "kube-api-access-m2nd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:41.122483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.122386    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2nd7\" (UniqueName: \"kubernetes.io/projected/90d90625-4204-4fec-8fb1-1237f358bd6a-kube-api-access-m2nd7\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:41.122483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.122414    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d90625-4204-4fec-8fb1-1237f358bd6a-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:41.296834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.296800    4851 generic.go:356] "Generic (PLEG): container finished" podID="90d90625-4204-4fec-8fb1-1237f358bd6a" containerID="a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec" exitCode=0
Nov 02 01:45:41.297311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.296844    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"
Nov 02 01:45:41.297311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.296883    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx" event={"ID":"90d90625-4204-4fec-8fb1-1237f358bd6a","Type":"ContainerDied","Data":"a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec"}
Nov 02 01:45:41.297311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.296938    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx" event={"ID":"90d90625-4204-4fec-8fb1-1237f358bd6a","Type":"ContainerDied","Data":"b2cddfa875dce68456b47a75693e8a8788cd3808628e5db187e973e423e238af"}
Nov 02 01:45:41.297311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.296966    4851 scope.go:122] "RemoveContainer" containerID="a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec"
Nov 02 01:45:41.299321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.299302    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"
Nov 02 01:45:41.299437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.299336    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z" event={"ID":"0f7739d3-8fb8-453e-8af5-345d8bd842e4","Type":"ContainerDied","Data":"91631e2e26e2f1ebdc1a7dc9a60d5854615b6015baf139657e938e009affecc2"}
Nov 02 01:45:41.310871 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.310846    4851 scope.go:122] "RemoveContainer" containerID="a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec"
Nov 02 01:45:41.311165 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:41.311143    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec\": container with ID starting with a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec not found: ID does not exist" containerID="a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec"
Nov 02 01:45:41.311237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.311173    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec"} err="failed to get container status \"a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec\": rpc error: code = NotFound desc = could not find container \"a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec\": container with ID starting with a895e418bcfb6f840791f8e38b0867e5b7728889bf23a6244a676b5838ea92ec not found: ID does not exist"
Nov 02 01:45:41.311237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.311191    4851 scope.go:122] "RemoveContainer" containerID="42a03eccb342b409f039e155501c3bdcbb7662883e12e09bd6a1d3281fda6773"
Nov 02 01:45:41.325326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.325304    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"]
Nov 02 01:45:41.339251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.339218    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f689bc-675c9556bc-5fh7z"]
Nov 02 01:45:41.386777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.386591    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"]
Nov 02 01:45:41.390372 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:41.390337    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f689bc-5fd6dcdb59-lrqfx"]
Nov 02 01:45:42.719214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:42.719178    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"]
Nov 02 01:45:42.723477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:42.723446    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bf1e55-f9dlp"]
Nov 02 01:45:42.932730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:42.932689    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f7739d3-8fb8-453e-8af5-345d8bd842e4" path="/var/lib/kubelet/pods/0f7739d3-8fb8-453e-8af5-345d8bd842e4/volumes"
Nov 02 01:45:42.934705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:42.934674    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90bc40f5-4ebc-4382-9182-40b1d463ba2e" path="/var/lib/kubelet/pods/90bc40f5-4ebc-4382-9182-40b1d463ba2e/volumes"
Nov 02 01:45:42.935586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:42.935562    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d90625-4204-4fec-8fb1-1237f358bd6a" path="/var/lib/kubelet/pods/90d90625-4204-4fec-8fb1-1237f358bd6a/volumes"
Nov 02 01:45:43.971784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:43.971740    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh" podUID="1e5b2266-6f15-44c8-b44c-e6288fae68ba" containerName="payload-processing" containerID="cri-o://15d44f96cd8984d2a2ca3cb9e73c13b267ca94b86cecd6507dabbff3b940136a" gracePeriod=25
Nov 02 01:45:43.993017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:43.992965    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf" podUID="e377a587-5a3a-46d6-a95e-b2a0decdf9c3" containerName="payload-pre-processing" containerID="cri-o://699ed7141a3af22f0dd36e5ff6a4b268ed855c77f15e32d7f40e5e2b84802d35" gracePeriod=25
Nov 02 01:45:44.565822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.565404    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:45:44.565822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.565477    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:45:44.572394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.572318    4851 generic.go:356] "Generic (PLEG): container finished" podID="1e5b2266-6f15-44c8-b44c-e6288fae68ba" containerID="15d44f96cd8984d2a2ca3cb9e73c13b267ca94b86cecd6507dabbff3b940136a" exitCode=0
Nov 02 01:45:44.572633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.572545    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh" event={"ID":"1e5b2266-6f15-44c8-b44c-e6288fae68ba","Type":"ContainerDied","Data":"15d44f96cd8984d2a2ca3cb9e73c13b267ca94b86cecd6507dabbff3b940136a"}
Nov 02 01:45:44.580095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.579883    4851 generic.go:356] "Generic (PLEG): container finished" podID="e377a587-5a3a-46d6-a95e-b2a0decdf9c3" containerID="699ed7141a3af22f0dd36e5ff6a4b268ed855c77f15e32d7f40e5e2b84802d35" exitCode=0
Nov 02 01:45:44.580095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.580032    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf" event={"ID":"e377a587-5a3a-46d6-a95e-b2a0decdf9c3","Type":"ContainerDied","Data":"699ed7141a3af22f0dd36e5ff6a4b268ed855c77f15e32d7f40e5e2b84802d35"}
Nov 02 01:45:44.596911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.592818    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"]
Nov 02 01:45:44.600531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.600505    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bbb385-wdjfv"]
Nov 02 01:45:44.631888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.631864    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:45:44.680921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.656522    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:45:44.785091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.785061    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-kube-api-access-nrtgb\" (UniqueName: \"kubernetes.io/projected/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-kube-api-access-nrtgb\") pod \"e377a587-5a3a-46d6-a95e-b2a0decdf9c3\" (UID: \"e377a587-5a3a-46d6-a95e-b2a0decdf9c3\") "
Nov 02 01:45:44.785285 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.785104    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-plugins-config-volume\") pod \"e377a587-5a3a-46d6-a95e-b2a0decdf9c3\" (UID: \"e377a587-5a3a-46d6-a95e-b2a0decdf9c3\") "
Nov 02 01:45:44.785382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.785360    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5b2266-6f15-44c8-b44c-e6288fae68ba-kube-api-access-djkwb\" (UniqueName: \"kubernetes.io/projected/1e5b2266-6f15-44c8-b44c-e6288fae68ba-kube-api-access-djkwb\") pod \"1e5b2266-6f15-44c8-b44c-e6288fae68ba\" (UID: \"1e5b2266-6f15-44c8-b44c-e6288fae68ba\") "
Nov 02 01:45:44.785443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.785417    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5b2266-6f15-44c8-b44c-e6288fae68ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5b2266-6f15-44c8-b44c-e6288fae68ba-plugins-config-volume\") pod \"1e5b2266-6f15-44c8-b44c-e6288fae68ba\" (UID: \"1e5b2266-6f15-44c8-b44c-e6288fae68ba\") "
Nov 02 01:45:44.785884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.785852    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-plugins-config-volume" pod "e377a587-5a3a-46d6-a95e-b2a0decdf9c3" (UID: "e377a587-5a3a-46d6-a95e-b2a0decdf9c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:44.785884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.785860    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5b2266-6f15-44c8-b44c-e6288fae68ba-plugins-config-volume" pod "1e5b2266-6f15-44c8-b44c-e6288fae68ba" (UID: "1e5b2266-6f15-44c8-b44c-e6288fae68ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:44.786127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.786106    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5b2266-6f15-44c8-b44c-e6288fae68ba-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:44.786187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.786132    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:44.787567 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.787542    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-kube-api-access-nrtgb" pod "e377a587-5a3a-46d6-a95e-b2a0decdf9c3" (UID: "e377a587-5a3a-46d6-a95e-b2a0decdf9c3"). InnerVolumeSpecName "kube-api-access-nrtgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:44.788155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.788136    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5b2266-6f15-44c8-b44c-e6288fae68ba-kube-api-access-djkwb" pod "1e5b2266-6f15-44c8-b44c-e6288fae68ba" (UID: "1e5b2266-6f15-44c8-b44c-e6288fae68ba"). InnerVolumeSpecName "kube-api-access-djkwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:44.881341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.881253    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32517f68-24d9-4b95-99bf-2cbcd7cfb599" path="/var/lib/kubelet/pods/32517f68-24d9-4b95-99bf-2cbcd7cfb599/volumes"
Nov 02 01:45:44.887210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.887184    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtgb\" (UniqueName: \"kubernetes.io/projected/e377a587-5a3a-46d6-a95e-b2a0decdf9c3-kube-api-access-nrtgb\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:44.887363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:44.887214    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djkwb\" (UniqueName: \"kubernetes.io/projected/1e5b2266-6f15-44c8-b44c-e6288fae68ba-kube-api-access-djkwb\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:45.600197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:45.597272    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf" event={"ID":"e377a587-5a3a-46d6-a95e-b2a0decdf9c3","Type":"ContainerDied","Data":"a7769dd7d5b654ebc42274fca456e673ab5760bb37f18d72ac35e2b03f351ade"}
Nov 02 01:45:45.600197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:45.597386    4851 scope.go:122] "RemoveContainer" containerID="699ed7141a3af22f0dd36e5ff6a4b268ed855c77f15e32d7f40e5e2b84802d35"
Nov 02 01:45:45.600197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:45.597608    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"
Nov 02 01:45:45.606003 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:45.605951    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh" event={"ID":"1e5b2266-6f15-44c8-b44c-e6288fae68ba","Type":"ContainerDied","Data":"a45167b0b94cbd0a67dbb26427f066031f51b149cf8f0e2d46696283cd2ce95a"}
Nov 02 01:45:45.606134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:45.606102    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"
Nov 02 01:45:45.620170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:45.620143    4851 scope.go:122] "RemoveContainer" containerID="15d44f96cd8984d2a2ca3cb9e73c13b267ca94b86cecd6507dabbff3b940136a"
Nov 02 01:45:45.625368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:45.625343    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"]
Nov 02 01:45:45.629416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:45.629388    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-bbb385-84c66489bc-dnglf"]
Nov 02 01:45:45.645188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:45.645155    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"]
Nov 02 01:45:45.651643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:45.651614    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-bbb385-79f4746847-l74lh"]
Nov 02 01:45:46.850370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:46.850337    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5b2266-6f15-44c8-b44c-e6288fae68ba" path="/var/lib/kubelet/pods/1e5b2266-6f15-44c8-b44c-e6288fae68ba/volumes"
Nov 02 01:45:46.850881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:46.850863    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e377a587-5a3a-46d6-a95e-b2a0decdf9c3" path="/var/lib/kubelet/pods/e377a587-5a3a-46d6-a95e-b2a0decdf9c3/volumes"
Nov 02 01:45:47.476667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.476628    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"]
Nov 02 01:45:47.477531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477513    4851 cpu_manager.go:414] "containerMap: removing container" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy"
Nov 02 01:45:47.477599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477534    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy"
Nov 02 01:45:47.477599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477560    4851 cpu_manager.go:414] "containerMap: removing container" podUID="1e5b2266-6f15-44c8-b44c-e6288fae68ba" containerName="payload-processing"
Nov 02 01:45:47.477599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477568    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5b2266-6f15-44c8-b44c-e6288fae68ba" containerName="payload-processing"
Nov 02 01:45:47.477599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477581    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerName="cleanup"
Nov 02 01:45:47.477599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477585    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d965b026-5f19-49bd-9cd4-9c8e2c16ecc0" containerName="cleanup"
Nov 02 01:45:47.477599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477592    4851 cpu_manager.go:414] "containerMap: removing container" podUID="90d90625-4204-4fec-8fb1-1237f358bd6a" containerName="payload-processing"
Nov 02 01:45:47.477599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477600    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d90625-4204-4fec-8fb1-1237f358bd6a" containerName="payload-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477620    4851 cpu_manager.go:414] "containerMap: removing container" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477627    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477639    4851 cpu_manager.go:414] "containerMap: removing container" podUID="e377a587-5a3a-46d6-a95e-b2a0decdf9c3" containerName="payload-pre-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477645    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e377a587-5a3a-46d6-a95e-b2a0decdf9c3" containerName="payload-pre-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477663    4851 cpu_manager.go:414] "containerMap: removing container" podUID="7d5a5b02-0b4a-4b54-a8e5-417b50f91a06" containerName="payload-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477671    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5a5b02-0b4a-4b54-a8e5-417b50f91a06" containerName="payload-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477681    4851 cpu_manager.go:414] "containerMap: removing container" podUID="32517f68-24d9-4b95-99bf-2cbcd7cfb599" containerName="revoke-keys"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477689    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32517f68-24d9-4b95-99bf-2cbcd7cfb599" containerName="revoke-keys"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477699    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0f7739d3-8fb8-453e-8af5-345d8bd842e4" containerName="payload-pre-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477703    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7739d3-8fb8-453e-8af5-345d8bd842e4" containerName="payload-pre-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477712    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477716    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477852    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbc6c42-07a0-412d-8a9d-36ce3c0686ab" containerName="cleanup"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477864    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477872    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5b2266-6f15-44c8-b44c-e6288fae68ba" containerName="payload-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477885    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d90625-4204-4fec-8fb1-1237f358bd6a" containerName="payload-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477897    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="e377a587-5a3a-46d6-a95e-b2a0decdf9c3" containerName="payload-pre-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477905    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5a5b02-0b4a-4b54-a8e5-417b50f91a06" containerName="payload-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477913    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="32517f68-24d9-4b95-99bf-2cbcd7cfb599" containerName="revoke-keys"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477921    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7739d3-8fb8-453e-8af5-345d8bd842e4" containerName="payload-pre-processing"
Nov 02 01:45:47.477942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.477932    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d430d5d-2307-41d0-a3fb-9576ce2b2c1e" containerName="istio-proxy"
Nov 02 01:45:47.503693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.503647    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"]
Nov 02 01:45:47.503886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.503771    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.509173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.509124    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c8d4a0-openshift-default-dockercfg-qcqst\""
Nov 02 01:45:47.647488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.646609    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-envoy\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.647488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.646706    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.647488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.646736    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.647488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.646777    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckrs7\" (UniqueName: \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-kube-api-access-ckrs7\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.647488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.646823    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-certs\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.647488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.646969    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-credential-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.647488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.647001    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-podinfo\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.647488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.647090    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-data\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.647488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.647153    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-token\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.783166    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-certs\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.783435    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-credential-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.783476    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-podinfo\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.783566    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-data\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.783630    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-token\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.783732    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-envoy\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.783804    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.783832    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.783862    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckrs7\" (UniqueName: \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-kube-api-access-ckrs7\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.784997    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-certs\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.794825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.785317    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-credential-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.809975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.802899    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-data\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.809975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.803597    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.809975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.803889    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.817866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.817808    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-podinfo\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.877380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.877251    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-envoy\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.877918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.877889    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckrs7\" (UniqueName: \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-kube-api-access-ckrs7\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.879494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:47.879449    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-token\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:47.896596 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:47.896564    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:45:48.152428 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:48.152354    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:48.589207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:48.589183    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"]
Nov 02 01:45:48.590570 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:48.590543    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa52a313_2e1d_4ee8_b80b_f89d0db88ff5.slice/crio-db2e6e38cc15b1f3fef1d40e07fa29dc12b599bf1fb74d9fdf64b772ce52c870 WatchSource:0}: Error finding container db2e6e38cc15b1f3fef1d40e07fa29dc12b599bf1fb74d9fdf64b772ce52c870: Status 404 returned error can't find the container with id db2e6e38cc15b1f3fef1d40e07fa29dc12b599bf1fb74d9fdf64b772ce52c870
Nov 02 01:45:48.593553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:48.593518    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:48.593674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:48.593583    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:48.593674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:48.593616    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:48.736928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:48.736854    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" event={"ID":"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5","Type":"ContainerStarted","Data":"db2e6e38cc15b1f3fef1d40e07fa29dc12b599bf1fb74d9fdf64b772ce52c870"}
Nov 02 01:45:49.877928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:49.768462    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" event={"ID":"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5","Type":"ContainerStarted","Data":"dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad"}
Nov 02 01:45:49.877928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:49.793819    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" podStartSLOduration=2.793803653 podStartE2EDuration="2.793803653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:49.791366137 +0000 UTC m=+1855.096298962" watchObservedRunningTime="2026-11-02 01:45:49.793803653 +0000 UTC m=+1855.098736479"
Nov 02 01:45:50.157713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.157615    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:50.160145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.160112    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 02 01:45:50.160351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.160175    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 02 01:45:50.787715 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.787634    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"]
Nov 02 01:45:50.800594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.787853    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy" containerID="cri-o://6988fe680d4b0c72953f4f9332ce85c07fc04ca5372042539f98ad51e5cfcdd3" gracePeriod=30
Nov 02 01:45:50.800594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.794891    4851 generic.go:356] "Generic (PLEG): container finished" podID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerID="4c4f2f98196832046611760edc552add6b53e824ea6b039127289953dbcdfbe7" exitCode=0
Nov 02 01:45:50.800594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.794971    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" event={"ID":"d486ac0b-a945-45aa-b47b-9ece667cb1f1","Type":"ContainerDied","Data":"4c4f2f98196832046611760edc552add6b53e824ea6b039127289953dbcdfbe7"}
Nov 02 01:45:50.800594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.795433    4851 scope.go:122] "RemoveContainer" containerID="4c4f2f98196832046611760edc552add6b53e824ea6b039127289953dbcdfbe7"
Nov 02 01:45:50.800594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.800387    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:50.800594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.800472    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:50.800594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.800540    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:50.804017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.803992    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"]
Nov 02 01:45:50.804209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.804177    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy" containerID="cri-o://bfb8c8f67febbeef3182da3ccd1c2e5089da76499b0bd9b3c72b0eec55d3cea4" gracePeriod=30
Nov 02 01:45:50.876075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.876037    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz" podUID="5eafdd80-7407-40d6-9579-cedb558b6411" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:45:50.908399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:50.908361    4851 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod119a2d02-b88d-4052-b81c-a43a441ff466"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod119a2d02-b88d-4052-b81c-a43a441ff466] : Timed out while waiting for systemd to remove kubepods-burstable-pod119a2d02_b88d_4052_b81c_a43a441ff466.slice"
Nov 02 01:45:50.908871 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:45:50.908427    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod119a2d02-b88d-4052-b81c-a43a441ff466] : unable to destroy cgroup paths for cgroup [kubepods burstable pod119a2d02-b88d-4052-b81c-a43a441ff466] : Timed out while waiting for systemd to remove kubepods-burstable-pod119a2d02_b88d_4052_b81c_a43a441ff466.slice" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn" podUID="119a2d02-b88d-4052-b81c-a43a441ff466"
Nov 02 01:45:51.202933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.202902    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 02 01:45:51.203118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.202962    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 02 01:45:51.287339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.286124    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"]
Nov 02 01:45:51.287774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.287528    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="016dfa35-82cc-4ab0-a117-b5f6a9bb2ead" containerName="istio-proxy"
Nov 02 01:45:51.294907 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.293172    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.299025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.298980    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"]
Nov 02 01:45:51.315069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.307119    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.315069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.307225    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-724ql\" (UniqueName: \"kubernetes.io/projected/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-kube-api-access-724ql\") pod \"maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.315069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.307329    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.377099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.377074    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:45:51.377335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.377098    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:45:51.408822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.408790    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.409009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.408891    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724ql\" (UniqueName: \"kubernetes.io/projected/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-kube-api-access-724ql\") pod \"maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.409128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.409083    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.409835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.409811    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.414152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.414124    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.421160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.421138    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-724ql\" (UniqueName: \"kubernetes.io/projected/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-kube-api-access-724ql\") pod \"maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.623172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.623087    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:51.848330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:51.830526    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" event={"ID":"d486ac0b-a945-45aa-b47b-9ece667cb1f1","Type":"ContainerStarted","Data":"d9804f6c59883965b505eb574f0f0aa4a4bb8974b4c4637856811ad53677f080"}
Nov 02 01:45:52.011470 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.009548    4851 generic.go:356] "Generic (PLEG): container finished" podID="70565ca1-c120-4b77-b307-a16604644193" containerID="94fc71b5e91b0a2ba607880b88ac0425b6eb41e9276a83763849a931901ce78f" exitCode=0
Nov 02 01:45:52.011470 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.009719    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"
Nov 02 01:45:52.011470 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.010389    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9" event={"ID":"70565ca1-c120-4b77-b307-a16604644193","Type":"ContainerDied","Data":"94fc71b5e91b0a2ba607880b88ac0425b6eb41e9276a83763849a931901ce78f"}
Nov 02 01:45:52.013684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.012682    4851 scope.go:122] "RemoveContainer" containerID="94fc71b5e91b0a2ba607880b88ac0425b6eb41e9276a83763849a931901ce78f"
Nov 02 01:45:52.015327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.014867    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:52.015327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.014947    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:52.015327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.014989    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:52.023795 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:45:52.023771    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77b6d46_8ca3_41d1_81a9_bab8317ba57b.slice/crio-2b84369e88514dca70c5fbce035effe1d1d58a1a49798315dad3d8e141310413 WatchSource:0}: Error finding container 2b84369e88514dca70c5fbce035effe1d1d58a1a49798315dad3d8e141310413: Status 404 returned error can't find the container with id 2b84369e88514dca70c5fbce035effe1d1d58a1a49798315dad3d8e141310413
Nov 02 01:45:52.026642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.026608    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"]
Nov 02 01:45:52.116876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.116827    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"]
Nov 02 01:45:52.365479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.354805    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 02 01:45:52.365479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.354871    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 02 01:45:52.491275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.491231    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"]
Nov 02 01:45:52.497608 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.497576    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893065-2sxsn"]
Nov 02 01:45:52.576322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.559334    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"]
Nov 02 01:45:52.602247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.602211    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:52.603812 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.603785    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"]
Nov 02 01:45:52.621381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.621324    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:45:52.622115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.622093    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 01:45:52.622248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.622131    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 01:45:52.740479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.738318    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbgj\" (UniqueName: \"kubernetes.io/projected/016b5ae3-8a02-4c61-ab83-55037e30cedf-kube-api-access-hhbgj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:52.740479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.738379    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:52.740479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.738531    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:52.846384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.842261    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:52.846384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.843038    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbgj\" (UniqueName: \"kubernetes.io/projected/016b5ae3-8a02-4c61-ab83-55037e30cedf-kube-api-access-hhbgj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:52.846384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.843085    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:52.846384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.843935    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:52.890854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.890741    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:52.891198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.891169    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbgj\" (UniqueName: \"kubernetes.io/projected/016b5ae3-8a02-4c61-ab83-55037e30cedf-kube-api-access-hhbgj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:52.900878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.900848    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119a2d02-b88d-4052-b81c-a43a441ff466" path="/var/lib/kubelet/pods/119a2d02-b88d-4052-b81c-a43a441ff466/volumes"
Nov 02 01:45:52.923280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:52.923259    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:53.025182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:53.025144    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q" event={"ID":"d77b6d46-8ca3-41d1-81a9-bab8317ba57b","Type":"ContainerStarted","Data":"7722b6954b95b1264b1dca275ae21eeb885a38f6f1439e69c1b2cc3cd8cb798b"}
Nov 02 01:45:53.025661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:53.025208    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q" event={"ID":"d77b6d46-8ca3-41d1-81a9-bab8317ba57b","Type":"ContainerStarted","Data":"2b84369e88514dca70c5fbce035effe1d1d58a1a49798315dad3d8e141310413"}
Nov 02 01:45:53.028274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:53.028248    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9" event={"ID":"70565ca1-c120-4b77-b307-a16604644193","Type":"ContainerStarted","Data":"cb934e169512e9cb87cd02fab0900214f262124f18539e1eca042a370312c892"}
Nov 02 01:45:53.028443 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:53.028335    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9" podUID="70565ca1-c120-4b77-b307-a16604644193" containerName="istio-proxy" containerID="cri-o://cb934e169512e9cb87cd02fab0900214f262124f18539e1eca042a370312c892" gracePeriod=30
Nov 02 01:45:53.103057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:53.095039    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q" podStartSLOduration=2.095018644 podStartE2EDuration="2.095018644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:53.048977311 +0000 UTC m=+1858.353910138" watchObservedRunningTime="2026-11-02 01:45:53.095018644 +0000 UTC m=+1858.399951470"
Nov 02 01:45:53.163632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:53.162311    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 02 01:45:53.163632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:53.162368    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 02 01:45:53.478540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:53.478507    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"]
Nov 02 01:45:53.621996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:53.621968    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 01:45:53.622168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:53.622020    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 01:45:54.091403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:54.088233    4851 generic.go:356] "Generic (PLEG): container finished" podID="d77b6d46-8ca3-41d1-81a9-bab8317ba57b" containerID="7722b6954b95b1264b1dca275ae21eeb885a38f6f1439e69c1b2cc3cd8cb798b" exitCode=0
Nov 02 01:45:54.091403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:54.088513    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q" event={"ID":"d77b6d46-8ca3-41d1-81a9-bab8317ba57b","Type":"ContainerDied","Data":"7722b6954b95b1264b1dca275ae21eeb885a38f6f1439e69c1b2cc3cd8cb798b"}
Nov 02 01:45:54.118346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:54.108502    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd" event={"ID":"016b5ae3-8a02-4c61-ab83-55037e30cedf","Type":"ContainerStarted","Data":"a10124dcd90bec9a41e741129d9117ea2b3f7c252b8f2a427bc9de3314c74edf"}
Nov 02 01:45:54.118346 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:54.108565    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd" event={"ID":"016b5ae3-8a02-4c61-ab83-55037e30cedf","Type":"ContainerStarted","Data":"46e67f2fe69504835057e1e6b5acc4713b3f75a0e871239eb421db3f098769a7"}
Nov 02 01:45:54.175327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:54.167798    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 02 01:45:54.175327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:54.167854    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 02 01:45:54.214589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:54.214528    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd" podStartSLOduration=2.214513076 podStartE2EDuration="2.214513076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:54.166813722 +0000 UTC m=+1859.471746549" watchObservedRunningTime="2026-11-02 01:45:54.214513076 +0000 UTC m=+1859.519445901"
Nov 02 01:45:54.622247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:54.622214    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 01:45:54.622470 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:54.622264    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 01:45:55.161446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.161410    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 02 01:45:55.162167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.162131    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 02 01:45:55.164254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.164234    4851 generic.go:356] "Generic (PLEG): container finished" podID="016b5ae3-8a02-4c61-ab83-55037e30cedf" containerID="a10124dcd90bec9a41e741129d9117ea2b3f7c252b8f2a427bc9de3314c74edf" exitCode=0
Nov 02 01:45:55.165000 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.164976    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd" event={"ID":"016b5ae3-8a02-4c61-ab83-55037e30cedf","Type":"ContainerDied","Data":"a10124dcd90bec9a41e741129d9117ea2b3f7c252b8f2a427bc9de3314c74edf"}
Nov 02 01:45:55.419845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.419776    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:55.569187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.569153    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-openshift-ca\") pod \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") "
Nov 02 01:45:55.569446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.569259    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-kube-api-access-724ql\" (UniqueName: \"kubernetes.io/projected/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-kube-api-access-724ql\") pod \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") "
Nov 02 01:45:55.569446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.569395    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-xks-ca\") pod \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\" (UID: \"d77b6d46-8ca3-41d1-81a9-bab8317ba57b\") "
Nov 02 01:45:55.569613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.569584    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-openshift-ca" pod "d77b6d46-8ca3-41d1-81a9-bab8317ba57b" (UID: "d77b6d46-8ca3-41d1-81a9-bab8317ba57b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:55.569947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.569926    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:55.571569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.571544    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-xks-ca" pod "d77b6d46-8ca3-41d1-81a9-bab8317ba57b" (UID: "d77b6d46-8ca3-41d1-81a9-bab8317ba57b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:45:55.571659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.571637    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-kube-api-access-724ql" pod "d77b6d46-8ca3-41d1-81a9-bab8317ba57b" (UID: "d77b6d46-8ca3-41d1-81a9-bab8317ba57b"). InnerVolumeSpecName "kube-api-access-724ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:55.622166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.622134    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 01:45:55.622354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.622189    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 01:45:55.684370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.683549    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:55.684370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.683594    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-724ql\" (UniqueName: \"kubernetes.io/projected/d77b6d46-8ca3-41d1-81a9-bab8317ba57b-kube-api-access-724ql\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:55.772359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:55.772106    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:45:56.184152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.184119    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:56.185768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.185747    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:56.188235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.188046    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:45:56.199509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.199483    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"
Nov 02 01:45:56.200445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.200060    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q" event={"ID":"d77b6d46-8ca3-41d1-81a9-bab8317ba57b","Type":"ContainerDied","Data":"2b84369e88514dca70c5fbce035effe1d1d58a1a49798315dad3d8e141310413"}
Nov 02 01:45:56.200445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.200102    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b84369e88514dca70c5fbce035effe1d1d58a1a49798315dad3d8e141310413"
Nov 02 01:45:56.202461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.201982    4851 generic.go:356] "Generic (PLEG): container finished" podID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerID="6988fe680d4b0c72953f4f9332ce85c07fc04ca5372042539f98ad51e5cfcdd3" exitCode=0
Nov 02 01:45:56.202461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.202119    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" event={"ID":"dca405bb-787c-44aa-83ae-cec31331ea1c","Type":"ContainerDied","Data":"6988fe680d4b0c72953f4f9332ce85c07fc04ca5372042539f98ad51e5cfcdd3"}
Nov 02 01:45:56.204382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.203940    4851 generic.go:356] "Generic (PLEG): container finished" podID="5eafdd80-7407-40d6-9579-cedb558b6411" containerID="679ca1dd80b9558dc896e5342c5463cf5ee3d884fe796199e92f43a1e5436601" exitCode=0
Nov 02 01:45:56.204382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.204062    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz" event={"ID":"5eafdd80-7407-40d6-9579-cedb558b6411","Type":"ContainerDied","Data":"679ca1dd80b9558dc896e5342c5463cf5ee3d884fe796199e92f43a1e5436601"}
Nov 02 01:45:56.208098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.206687    4851 generic.go:356] "Generic (PLEG): container finished" podID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerID="bfb8c8f67febbeef3182da3ccd1c2e5089da76499b0bd9b3c72b0eec55d3cea4" exitCode=0
Nov 02 01:45:56.208098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.206861    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" event={"ID":"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b","Type":"ContainerDied","Data":"bfb8c8f67febbeef3182da3ccd1c2e5089da76499b0bd9b3c72b0eec55d3cea4"}
Nov 02 01:45:56.208098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.206892    4851 scope.go:122] "RemoveContainer" containerID="f1769e89b2b8bab680c9f8909458a125a26c2926c597f4cb936a09869cf1c68c"
Nov 02 01:45:56.222585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.222557    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:45:56.292844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.292812    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eafdd80-7407-40d6-9579-cedb558b6411-kube-api-access-khlqm\" (UniqueName: \"kubernetes.io/projected/5eafdd80-7407-40d6-9579-cedb558b6411-kube-api-access-khlqm\") pod \"5eafdd80-7407-40d6-9579-cedb558b6411\" (UID: \"5eafdd80-7407-40d6-9579-cedb558b6411\") "
Nov 02 01:45:56.294946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.294922    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eafdd80-7407-40d6-9579-cedb558b6411-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eafdd80-7407-40d6-9579-cedb558b6411-plugins-config-volume\") pod \"5eafdd80-7407-40d6-9579-cedb558b6411\" (UID: \"5eafdd80-7407-40d6-9579-cedb558b6411\") "
Nov 02 01:45:56.298750 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.298726    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eafdd80-7407-40d6-9579-cedb558b6411-plugins-config-volume" pod "5eafdd80-7407-40d6-9579-cedb558b6411" (UID: "5eafdd80-7407-40d6-9579-cedb558b6411"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:56.310026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.309327    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:45:56.310525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.310484    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eafdd80-7407-40d6-9579-cedb558b6411-kube-api-access-khlqm" pod "5eafdd80-7407-40d6-9579-cedb558b6411" (UID: "5eafdd80-7407-40d6-9579-cedb558b6411"). InnerVolumeSpecName "kube-api-access-khlqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:56.397187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.397153    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-data\") pod \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") "
Nov 02 01:45:56.397448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.397428    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-credential-socket\") pod \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") "
Nov 02 01:45:56.397581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.397565    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-envoy\") pod \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") "
Nov 02 01:45:56.397793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.397697    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-kube-api-access-xx4wj\" (UniqueName: \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-kube-api-access-xx4wj\") pod \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") "
Nov 02 01:45:56.397992 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.397976    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-podinfo\") pod \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") "
Nov 02 01:45:56.398185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.398169    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-token\" (UniqueName: \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-token\") pod \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") "
Nov 02 01:45:56.398398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.398379    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istiod-ca-cert\") pod \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") "
Nov 02 01:45:56.398623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.398605    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-certs\") pod \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") "
Nov 02 01:45:56.398768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.398745    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-socket\") pod \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\" (UID: \"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b\") "
Nov 02 01:45:56.399968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.399868    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-credential-socket" pod "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" (UID: "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:56.400490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.400180    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-data" pod "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" (UID: "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:56.401009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.400912    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istiod-ca-cert" pod "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" (UID: "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:56.403027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.402891    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-certs" pod "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" (UID: "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:56.404445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.404330    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-socket" pod "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" (UID: "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:56.405673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.405542    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khlqm\" (UniqueName: \"kubernetes.io/projected/5eafdd80-7407-40d6-9579-cedb558b6411-kube-api-access-khlqm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.405938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.405921    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.406060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.406048    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.406157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.406146    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.406247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.406235    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eafdd80-7407-40d6-9579-cedb558b6411-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.406395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.406382    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.406501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.406489    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.406584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.406342    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-envoy" pod "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" (UID: "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:56.407594 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.407571    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-kube-api-access-xx4wj" pod "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" (UID: "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b"). InnerVolumeSpecName "kube-api-access-xx4wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:56.407757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.407573    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-podinfo" pod "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" (UID: "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:45:56.410517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.410244    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-token" pod "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" (UID: "9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:56.479065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.479036    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.507027    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-kube-api-access-96fth\" (UniqueName: \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-kube-api-access-96fth\") pod \"dca405bb-787c-44aa-83ae-cec31331ea1c\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") "
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.507094    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-credential-socket\") pod \"dca405bb-787c-44aa-83ae-cec31331ea1c\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") "
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.507168    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-token\" (UniqueName: \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-token\") pod \"dca405bb-787c-44aa-83ae-cec31331ea1c\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") "
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.507201    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-envoy\") pod \"dca405bb-787c-44aa-83ae-cec31331ea1c\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") "
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.507275    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-certs\") pod \"dca405bb-787c-44aa-83ae-cec31331ea1c\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") "
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.507357    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-podinfo\") pod \"dca405bb-787c-44aa-83ae-cec31331ea1c\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") "
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.507512    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca405bb-787c-44aa-83ae-cec31331ea1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca405bb-787c-44aa-83ae-cec31331ea1c-istiod-ca-cert\") pod \"dca405bb-787c-44aa-83ae-cec31331ea1c\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") "
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.507555    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-data\") pod \"dca405bb-787c-44aa-83ae-cec31331ea1c\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") "
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.507662    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-socket\") pod \"dca405bb-787c-44aa-83ae-cec31331ea1c\" (UID: \"dca405bb-787c-44aa-83ae-cec31331ea1c\") "
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.508228    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.508247    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx4wj\" (UniqueName: \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-kube-api-access-xx4wj\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.508262    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.508496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.508275    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.512345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.510739    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca405bb-787c-44aa-83ae-cec31331ea1c-istiod-ca-cert" pod "dca405bb-787c-44aa-83ae-cec31331ea1c" (UID: "dca405bb-787c-44aa-83ae-cec31331ea1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:56.512345 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.510971    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-data" pod "dca405bb-787c-44aa-83ae-cec31331ea1c" (UID: "dca405bb-787c-44aa-83ae-cec31331ea1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:56.513153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.512622    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-socket" pod "dca405bb-787c-44aa-83ae-cec31331ea1c" (UID: "dca405bb-787c-44aa-83ae-cec31331ea1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:56.517461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.517422    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-credential-socket" pod "dca405bb-787c-44aa-83ae-cec31331ea1c" (UID: "dca405bb-787c-44aa-83ae-cec31331ea1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:56.518334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.518164    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-certs" pod "dca405bb-787c-44aa-83ae-cec31331ea1c" (UID: "dca405bb-787c-44aa-83ae-cec31331ea1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:56.520503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.520451    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-kube-api-access-96fth" pod "dca405bb-787c-44aa-83ae-cec31331ea1c" (UID: "dca405bb-787c-44aa-83ae-cec31331ea1c"). InnerVolumeSpecName "kube-api-access-96fth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:56.524045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.524023    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:56.532110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.532070    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-envoy" pod "dca405bb-787c-44aa-83ae-cec31331ea1c" (UID: "dca405bb-787c-44aa-83ae-cec31331ea1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:56.552362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.549440    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-podinfo" pod "dca405bb-787c-44aa-83ae-cec31331ea1c" (UID: "dca405bb-787c-44aa-83ae-cec31331ea1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:45:56.566036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.565622    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-token" pod "dca405bb-787c-44aa-83ae-cec31331ea1c" (UID: "dca405bb-787c-44aa-83ae-cec31331ea1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:56.609341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.608836    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-xks-ca\") pod \"016b5ae3-8a02-4c61-ab83-55037e30cedf\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") "
Nov 02 01:45:56.609341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.609069    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016b5ae3-8a02-4c61-ab83-55037e30cedf-kube-api-access-hhbgj\" (UniqueName: \"kubernetes.io/projected/016b5ae3-8a02-4c61-ab83-55037e30cedf-kube-api-access-hhbgj\") pod \"016b5ae3-8a02-4c61-ab83-55037e30cedf\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") "
Nov 02 01:45:56.609341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.609131    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-openshift-ca\") pod \"016b5ae3-8a02-4c61-ab83-55037e30cedf\" (UID: \"016b5ae3-8a02-4c61-ab83-55037e30cedf\") "
Nov 02 01:45:56.610202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.609934    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-openshift-ca" pod "016b5ae3-8a02-4c61-ab83-55037e30cedf" (UID: "016b5ae3-8a02-4c61-ab83-55037e30cedf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:56.610438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.610414    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96fth\" (UniqueName: \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-kube-api-access-96fth\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.610555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.610443    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.610555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.610458    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.610555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.610472    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.610555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.610485    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.610555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.610498    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.610555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.610511    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.610555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.610524    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca405bb-787c-44aa-83ae-cec31331ea1c-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.610555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.610538    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.610555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.610552    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca405bb-787c-44aa-83ae-cec31331ea1c-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.615138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.615070    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-xks-ca" pod "016b5ae3-8a02-4c61-ab83-55037e30cedf" (UID: "016b5ae3-8a02-4c61-ab83-55037e30cedf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:45:56.615138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.615113    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016b5ae3-8a02-4c61-ab83-55037e30cedf-kube-api-access-hhbgj" pod "016b5ae3-8a02-4c61-ab83-55037e30cedf" (UID: "016b5ae3-8a02-4c61-ab83-55037e30cedf"). InnerVolumeSpecName "kube-api-access-hhbgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:56.665253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.665181    4851 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 01:45:56.665995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.665905    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 01:45:56.711504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.711469    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhbgj\" (UniqueName: \"kubernetes.io/projected/016b5ae3-8a02-4c61-ab83-55037e30cedf-kube-api-access-hhbgj\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:56.711504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:56.711504    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016b5ae3-8a02-4c61-ab83-55037e30cedf-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:57.218242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.218209    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"
Nov 02 01:45:57.218747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.218235    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp" event={"ID":"dca405bb-787c-44aa-83ae-cec31331ea1c","Type":"ContainerDied","Data":"4a4bfe3c4535e87f566a362bfddc74a182c7534faef0ee391bcbeeedffe042fb"}
Nov 02 01:45:57.218747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.218325    4851 scope.go:122] "RemoveContainer" containerID="6988fe680d4b0c72953f4f9332ce85c07fc04ca5372042539f98ad51e5cfcdd3"
Nov 02 01:45:57.220339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.220002    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"
Nov 02 01:45:57.220339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.220029    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd" event={"ID":"016b5ae3-8a02-4c61-ab83-55037e30cedf","Type":"ContainerDied","Data":"46e67f2fe69504835057e1e6b5acc4713b3f75a0e871239eb421db3f098769a7"}
Nov 02 01:45:57.220339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.220079    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46e67f2fe69504835057e1e6b5acc4713b3f75a0e871239eb421db3f098769a7"
Nov 02 01:45:57.222090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.222063    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"
Nov 02 01:45:57.222199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.222103    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz" event={"ID":"5eafdd80-7407-40d6-9579-cedb558b6411","Type":"ContainerDied","Data":"3fd8d15f4bcfe32ed06895e7138a854ae4e350221a96d838fd1e3739e0bb88e0"}
Nov 02 01:45:57.224466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.224443    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8" event={"ID":"9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b","Type":"ContainerDied","Data":"db53cebf2866c0f98c5a9a345814809e86996420f32f60f5cec5a747da80f1af"}
Nov 02 01:45:57.224867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.224853    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"
Nov 02 01:45:57.265313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.259866    4851 scope.go:122] "RemoveContainer" containerID="679ca1dd80b9558dc896e5342c5463cf5ee3d884fe796199e92f43a1e5436601"
Nov 02 01:45:57.269313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.265980    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"]
Nov 02 01:45:57.272592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.272572    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-vjbm8"]
Nov 02 01:45:57.277200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.277169    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"]
Nov 02 01:45:57.281592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.281523    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf1e55-74c866bcb7-jcgsz"]
Nov 02 01:45:57.282675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.282657    4851 scope.go:122] "RemoveContainer" containerID="bfb8c8f67febbeef3182da3ccd1c2e5089da76499b0bd9b3c72b0eec55d3cea4"
Nov 02 01:45:57.288516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.288490    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"]
Nov 02 01:45:57.292725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.292699    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-xhcwp"]
Nov 02 01:45:57.685926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.685894    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:45:57.686592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.686565    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:45:57.689526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:57.689500    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:45:58.242563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.242530    4851 generic.go:356] "Generic (PLEG): container finished" podID="70565ca1-c120-4b77-b307-a16604644193" containerID="cb934e169512e9cb87cd02fab0900214f262124f18539e1eca042a370312c892" exitCode=0
Nov 02 01:45:58.242927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.242600    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9" event={"ID":"70565ca1-c120-4b77-b307-a16604644193","Type":"ContainerDied","Data":"cb934e169512e9cb87cd02fab0900214f262124f18539e1eca042a370312c892"}
Nov 02 01:45:58.242927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.242653    4851 scope.go:122] "RemoveContainer" containerID="94fc71b5e91b0a2ba607880b88ac0425b6eb41e9276a83763849a931901ce78f"
Nov 02 01:45:58.264796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.264775    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:45:58.335002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.334904    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-credential-socket\") pod \"70565ca1-c120-4b77-b307-a16604644193\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") "
Nov 02 01:45:58.335002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.334975    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-data\") pod \"70565ca1-c120-4b77-b307-a16604644193\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") "
Nov 02 01:45:58.335249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335047    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-envoy\") pod \"70565ca1-c120-4b77-b307-a16604644193\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") "
Nov 02 01:45:58.335249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335085    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70565ca1-c120-4b77-b307-a16604644193-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70565ca1-c120-4b77-b307-a16604644193-istio-podinfo\") pod \"70565ca1-c120-4b77-b307-a16604644193\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") "
Nov 02 01:45:58.335249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335190    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70565ca1-c120-4b77-b307-a16604644193-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70565ca1-c120-4b77-b307-a16604644193-istiod-ca-cert\") pod \"70565ca1-c120-4b77-b307-a16604644193\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") "
Nov 02 01:45:58.335452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335336    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-credential-socket" pod "70565ca1-c120-4b77-b307-a16604644193" (UID: "70565ca1-c120-4b77-b307-a16604644193"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:58.335452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335403    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-certs\") pod \"70565ca1-c120-4b77-b307-a16604644193\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") "
Nov 02 01:45:58.335452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335431    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-data" pod "70565ca1-c120-4b77-b307-a16604644193" (UID: "70565ca1-c120-4b77-b307-a16604644193"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:58.335452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335443    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-kube-api-access-qsr4v\" (UniqueName: \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-kube-api-access-qsr4v\") pod \"70565ca1-c120-4b77-b307-a16604644193\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") "
Nov 02 01:45:58.335658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335483    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-istio-token\" (UniqueName: \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-istio-token\") pod \"70565ca1-c120-4b77-b307-a16604644193\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") "
Nov 02 01:45:58.335658 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335546    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-socket\") pod \"70565ca1-c120-4b77-b307-a16604644193\" (UID: \"70565ca1-c120-4b77-b307-a16604644193\") "
Nov 02 01:45:58.335787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335701    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70565ca1-c120-4b77-b307-a16604644193-istiod-ca-cert" pod "70565ca1-c120-4b77-b307-a16604644193" (UID: "70565ca1-c120-4b77-b307-a16604644193"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:58.335787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.335713    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-certs" pod "70565ca1-c120-4b77-b307-a16604644193" (UID: "70565ca1-c120-4b77-b307-a16604644193"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:58.336835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.336017    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-socket" pod "70565ca1-c120-4b77-b307-a16604644193" (UID: "70565ca1-c120-4b77-b307-a16604644193"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:58.336835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.336451    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70565ca1-c120-4b77-b307-a16604644193-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:58.336835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.336476    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:58.336835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.336492    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:58.336835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.336508    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:58.336835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.336523    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:58.338262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.338228    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-kube-api-access-qsr4v" pod "70565ca1-c120-4b77-b307-a16604644193" (UID: "70565ca1-c120-4b77-b307-a16604644193"). InnerVolumeSpecName "kube-api-access-qsr4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:58.338414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.338388    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70565ca1-c120-4b77-b307-a16604644193-istio-podinfo" pod "70565ca1-c120-4b77-b307-a16604644193" (UID: "70565ca1-c120-4b77-b307-a16604644193"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:45:58.338595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.338572    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-istio-token" pod "70565ca1-c120-4b77-b307-a16604644193" (UID: "70565ca1-c120-4b77-b307-a16604644193"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:58.338902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.338879    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-envoy" pod "70565ca1-c120-4b77-b307-a16604644193" (UID: "70565ca1-c120-4b77-b307-a16604644193"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:45:58.437804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.437750    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70565ca1-c120-4b77-b307-a16604644193-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:58.437804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.437796    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70565ca1-c120-4b77-b307-a16604644193-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:58.437804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.437808    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsr4v\" (UniqueName: \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-kube-api-access-qsr4v\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:58.437804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.437817    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70565ca1-c120-4b77-b307-a16604644193-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:45:58.850138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.850096    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eafdd80-7407-40d6-9579-cedb558b6411" path="/var/lib/kubelet/pods/5eafdd80-7407-40d6-9579-cedb558b6411/volumes"
Nov 02 01:45:58.850592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.850574    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" path="/var/lib/kubelet/pods/9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b/volumes"
Nov 02 01:45:58.851019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:58.851004    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" path="/var/lib/kubelet/pods/dca405bb-787c-44aa-83ae-cec31331ea1c/volumes"
Nov 02 01:45:59.253263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.253230    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"
Nov 02 01:45:59.253787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.253266    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9" event={"ID":"70565ca1-c120-4b77-b307-a16604644193","Type":"ContainerDied","Data":"ef27e4f84a08eaaa81e1b5015e6ac7a095a95c5e9a221420342cf84c09a460c7"}
Nov 02 01:45:59.253787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.253346    4851 scope.go:122] "RemoveContainer" containerID="cb934e169512e9cb87cd02fab0900214f262124f18539e1eca042a370312c892"
Nov 02 01:45:59.295528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.295499    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"]
Nov 02 01:45:59.299174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.299146    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bbb385-openshift-default-75fbdd7797-4tpw9"]
Nov 02 01:45:59.523305 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.523206    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"]
Nov 02 01:45:59.523989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.523967    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5eafdd80-7407-40d6-9579-cedb558b6411" containerName="payload-pre-processing"
Nov 02 01:45:59.523989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.523988    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eafdd80-7407-40d6-9579-cedb558b6411" containerName="payload-pre-processing"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524014    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524019    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524026    4851 cpu_manager.go:414] "containerMap: removing container" podUID="70565ca1-c120-4b77-b307-a16604644193" containerName="istio-proxy"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524031    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70565ca1-c120-4b77-b307-a16604644193" containerName="istio-proxy"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524043    4851 cpu_manager.go:414] "containerMap: removing container" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524047    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524058    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524062    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524069    4851 cpu_manager.go:414] "containerMap: removing container" podUID="016b5ae3-8a02-4c61-ab83-55037e30cedf" containerName="revoke-keys"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524073    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016b5ae3-8a02-4c61-ab83-55037e30cedf" containerName="revoke-keys"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524093    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d77b6d46-8ca3-41d1-81a9-bab8317ba57b" containerName="revoke-keys"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524098    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77b6d46-8ca3-41d1-81a9-bab8317ba57b" containerName="revoke-keys"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524186    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eafdd80-7407-40d6-9579-cedb558b6411" containerName="payload-pre-processing"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524198    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy"
Nov 02 01:45:59.524196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524205    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="70565ca1-c120-4b77-b307-a16604644193" containerName="istio-proxy"
Nov 02 01:45:59.525511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524210    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca405bb-787c-44aa-83ae-cec31331ea1c" containerName="istio-proxy"
Nov 02 01:45:59.525511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524216    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae956ce-3eb3-44b7-a7c1-487ec5a5d21b" containerName="istio-proxy"
Nov 02 01:45:59.525511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524220    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="016b5ae3-8a02-4c61-ab83-55037e30cedf" containerName="revoke-keys"
Nov 02 01:45:59.525511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.524228    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77b6d46-8ca3-41d1-81a9-bab8317ba57b" containerName="revoke-keys"
Nov 02 01:45:59.529962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.529936    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.534262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.534238    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d4a0\"/\"auth-test-model-c8d4a0-kserve-self-signed-certs\""
Nov 02 01:45:59.534583 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.534565    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d4a0\"/\"default-dockercfg-h7kg6\""
Nov 02 01:45:59.537100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.537076    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"]
Nov 02 01:45:59.545668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.544931    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d4a0\"/\"kube-root-ca.crt\""
Nov 02 01:45:59.545668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.545032    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d4a0\"/\"openshift-service-ca.crt\""
Nov 02 01:45:59.654445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.654413    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tls-certs\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.654623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.654527    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-model-cache\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.654623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.654562    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tmp-dir\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.654699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.654658    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-dshm\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.654773 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.654743    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-home\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.654844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.654829    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x4jh\" (UniqueName: \"kubernetes.io/projected/e713d800-8e9d-4d87-8b1d-8d039c1d379b-kube-api-access-5x4jh\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.756253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.756203    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-dshm\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.756590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.756563    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-home\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.756907 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.756885    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x4jh\" (UniqueName: \"kubernetes.io/projected/e713d800-8e9d-4d87-8b1d-8d039c1d379b-kube-api-access-5x4jh\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.757226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.757206    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tls-certs\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.757414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.757393    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-model-cache\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.757546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.757530    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tmp-dir\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.767442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.758143    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tmp-dir\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.767442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.766355    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-model-cache\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.767442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.766584    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-home\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.776352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.774006    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-dshm\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.776352 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.774738    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tls-certs\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.794200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.793326    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x4jh\" (UniqueName: \"kubernetes.io/projected/e713d800-8e9d-4d87-8b1d-8d039c1d379b-kube-api-access-5x4jh\") pod \"auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") " pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:45:59.850718 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:45:59.850681    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:46:00.142651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:00.142623    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"]
Nov 02 01:46:00.143273 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:46:00.143250    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode713d800_8e9d_4d87_8b1d_8d039c1d379b.slice/crio-a364ac649c538e53a06e38840c698bc5fa688ff970bbb93ba57d1221c18fca1d WatchSource:0}: Error finding container a364ac649c538e53a06e38840c698bc5fa688ff970bbb93ba57d1221c18fca1d: Status 404 returned error can't find the container with id a364ac649c538e53a06e38840c698bc5fa688ff970bbb93ba57d1221c18fca1d
Nov 02 01:46:00.270366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:00.270320    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l" event={"ID":"e713d800-8e9d-4d87-8b1d-8d039c1d379b","Type":"ContainerStarted","Data":"cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2"}
Nov 02 01:46:00.270809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:00.270370    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l" event={"ID":"e713d800-8e9d-4d87-8b1d-8d039c1d379b","Type":"ContainerStarted","Data":"a364ac649c538e53a06e38840c698bc5fa688ff970bbb93ba57d1221c18fca1d"}
Nov 02 01:46:00.289756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:00.289714    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l" podStartSLOduration=1.2896992250000001 podStartE2EDuration="1.289699225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:00.288693905 +0000 UTC m=+1865.593626749" watchObservedRunningTime="2026-11-02 01:46:00.289699225 +0000 UTC m=+1865.594632050"
Nov 02 01:46:00.848677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:00.848643    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70565ca1-c120-4b77-b307-a16604644193" path="/var/lib/kubelet/pods/70565ca1-c120-4b77-b307-a16604644193/volumes"
Nov 02 01:46:00.851065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:00.851044    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:46:01.442994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:01.442962    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"]
Nov 02 01:46:01.454313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:01.454260    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"]
Nov 02 01:46:02.672015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:02.671717    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"]
Nov 02 01:46:05.057613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:05.057571    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"]
Nov 02 01:46:05.062967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:05.062938    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-00f1da-rcghd"]
Nov 02 01:46:06.446326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:06.445398    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5" podUID="fe26ff80-ecf0-4541-adc0-56877c9c009d" containerName="payload-processing" containerID="cri-o://cd55ebf79dbc04e80f5fa0dbff509bc063845c65759293d3ffa8a08e0c65ee71" gracePeriod=25
Nov 02 01:46:06.457462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:06.457386    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf" podUID="b3daa532-ae71-4a64-a684-59d4fbb22088" containerName="payload-pre-processing" containerID="cri-o://6a74d7dcc7cd4f0112f3400d171f6b96d0fa7b6fb9031a72fc9b9e160639b5f5" gracePeriod=25
Nov 02 01:46:06.872993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:06.871092    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016b5ae3-8a02-4c61-ab83-55037e30cedf" path="/var/lib/kubelet/pods/016b5ae3-8a02-4c61-ab83-55037e30cedf/volumes"
Nov 02 01:46:07.672526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:07.672479    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh" podUID="bb9669fc-e378-4a4b-aea5-ea97c9b71882" containerName="payload-pre-processing" containerID="cri-o://f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be" gracePeriod=25
Nov 02 01:46:07.683926 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:46:07.683895    4851 logging.go:55] [core] [Channel #6103 SubChannel #6104]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.179:50052", ServerName: "10.132.0.179:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.179:50052: connect: connection refused"
Nov 02 01:46:08.089935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.089906    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:46:08.195516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.195485    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb9669fc-e378-4a4b-aea5-ea97c9b71882-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9669fc-e378-4a4b-aea5-ea97c9b71882-plugins-config-volume\") pod \"bb9669fc-e378-4a4b-aea5-ea97c9b71882\" (UID: \"bb9669fc-e378-4a4b-aea5-ea97c9b71882\") "
Nov 02 01:46:08.195699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.195607    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9669fc-e378-4a4b-aea5-ea97c9b71882-kube-api-access-fm98p\" (UniqueName: \"kubernetes.io/projected/bb9669fc-e378-4a4b-aea5-ea97c9b71882-kube-api-access-fm98p\") pod \"bb9669fc-e378-4a4b-aea5-ea97c9b71882\" (UID: \"bb9669fc-e378-4a4b-aea5-ea97c9b71882\") "
Nov 02 01:46:08.195887 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.195863    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9669fc-e378-4a4b-aea5-ea97c9b71882-plugins-config-volume" pod "bb9669fc-e378-4a4b-aea5-ea97c9b71882" (UID: "bb9669fc-e378-4a4b-aea5-ea97c9b71882"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:08.196219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.196195    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9669fc-e378-4a4b-aea5-ea97c9b71882-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:08.197881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.197856    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9669fc-e378-4a4b-aea5-ea97c9b71882-kube-api-access-fm98p" pod "bb9669fc-e378-4a4b-aea5-ea97c9b71882" (UID: "bb9669fc-e378-4a4b-aea5-ea97c9b71882"). InnerVolumeSpecName "kube-api-access-fm98p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:08.297771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.297731    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm98p\" (UniqueName: \"kubernetes.io/projected/bb9669fc-e378-4a4b-aea5-ea97c9b71882-kube-api-access-fm98p\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:08.458936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.458894    4851 generic.go:356] "Generic (PLEG): container finished" podID="bb9669fc-e378-4a4b-aea5-ea97c9b71882" containerID="f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be" exitCode=0
Nov 02 01:46:08.458936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.458936    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"
Nov 02 01:46:08.459234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.458980    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh" event={"ID":"bb9669fc-e378-4a4b-aea5-ea97c9b71882","Type":"ContainerDied","Data":"f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be"}
Nov 02 01:46:08.459234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.459049    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh" event={"ID":"bb9669fc-e378-4a4b-aea5-ea97c9b71882","Type":"ContainerDied","Data":"d41e89e3d876899d46f195221169f73f101e23ab6a0f6e8d784a02cbc0c52239"}
Nov 02 01:46:08.459234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.459082    4851 scope.go:122] "RemoveContainer" containerID="f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be"
Nov 02 01:46:08.471903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.471883    4851 scope.go:122] "RemoveContainer" containerID="f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be"
Nov 02 01:46:08.472206 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:46:08.472183    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be\": container with ID starting with f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be not found: ID does not exist" containerID="f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be"
Nov 02 01:46:08.472275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.472220    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be"} err="failed to get container status \"f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be\": rpc error: code = NotFound desc = could not find container \"f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be\": container with ID starting with f8c929a309293238cbf480de48c2e1eb5e102497b6ed3d7b772c56efb9ce25be not found: ID does not exist"
Nov 02 01:46:08.484115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.484082    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"]
Nov 02 01:46:08.487835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.487811    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh"]
Nov 02 01:46:08.691324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.686451    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-00f1da-688cfc9cff-vhmkh" podUID="bb9669fc-e378-4a4b-aea5-ea97c9b71882" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.179:50052\" within 1s: context deadline exceeded"
Nov 02 01:46:08.691324 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:46:08.686900    4851 logging.go:55] [core] [Channel #6103 SubChannel #6104]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.179:50052", ServerName: "10.132.0.179:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.179:50052: operation was canceled"
Nov 02 01:46:08.870432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:08.870346    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb9669fc-e378-4a4b-aea5-ea97c9b71882" path="/var/lib/kubelet/pods/bb9669fc-e378-4a4b-aea5-ea97c9b71882/volumes"
Nov 02 01:46:09.857548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:09.857497    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:46:09.871892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:09.871865    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:46:10.055384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:10.054339    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"]
Nov 02 01:46:10.075317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:10.071818    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-12e7cb-t9f2q"]
Nov 02 01:46:10.560318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:10.558625    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:46:10.864481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:10.864393    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77b6d46-8ca3-41d1-81a9-bab8317ba57b" path="/var/lib/kubelet/pods/d77b6d46-8ca3-41d1-81a9-bab8317ba57b/volumes"
Nov 02 01:46:11.855141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:11.855105    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf" podUID="b3daa532-ae71-4a64-a684-59d4fbb22088" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:46:11.855141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:11.855135    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5" podUID="fe26ff80-ecf0-4541-adc0-56877c9c009d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:46:12.492889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:12.492830    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"]
Nov 02 01:46:12.493470 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:12.493149    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy" containerID="cri-o://e4e735e6a235d02542919c015c294de6e7f33ceef0038a69157655bf471e4628" gracePeriod=30
Nov 02 01:46:16.606041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:16.606003    4851 generic.go:356] "Generic (PLEG): container finished" podID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerID="425c1575a0d8a761f7b21ee2d9b69e7b07dddbcc0c7656a5454786f8eab9141f" exitCode=0
Nov 02 01:46:16.606532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:16.606074    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" event={"ID":"5377e2b2-cc37-4884-96a0-9a5e2628d968","Type":"ContainerDied","Data":"425c1575a0d8a761f7b21ee2d9b69e7b07dddbcc0c7656a5454786f8eab9141f"}
Nov 02 01:46:16.606577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:16.606567    4851 scope.go:122] "RemoveContainer" containerID="425c1575a0d8a761f7b21ee2d9b69e7b07dddbcc0c7656a5454786f8eab9141f"
Nov 02 01:46:16.608280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:16.608246    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:46:16.608423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:16.608341    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:46:16.608423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:16.608373    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:46:17.672052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:17.672014    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" event={"ID":"5377e2b2-cc37-4884-96a0-9a5e2628d968","Type":"ContainerStarted","Data":"6a16f6aa768b57d1737e5f51a12cf40e31d00f16f8faaa7e2ba86c1610b9216c"}
Nov 02 01:46:17.675343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:17.675263    4851 generic.go:356] "Generic (PLEG): container finished" podID="da59dac5-1239-4ee9-b7af-f949e9891687" containerID="e4e735e6a235d02542919c015c294de6e7f33ceef0038a69157655bf471e4628" exitCode=0
Nov 02 01:46:17.675584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:17.675360    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" event={"ID":"da59dac5-1239-4ee9-b7af-f949e9891687","Type":"ContainerDied","Data":"e4e735e6a235d02542919c015c294de6e7f33ceef0038a69157655bf471e4628"}
Nov 02 01:46:17.675672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:17.675622    4851 scope.go:122] "RemoveContainer" containerID="b9a724f46926be34a1037c594310b576b54f2ab11d96c0bb0170a59bb440c0fb"
Nov 02 01:46:17.699497 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:46:17.699455    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:46:18.228305 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.228260    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:46:18.228480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.228467    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 01:46:18.228524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.228505    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 01:46:18.392908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.392884    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:46:18.428774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.428739    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-kube-api-access-s6pck\" (UniqueName: \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-kube-api-access-s6pck\") pod \"da59dac5-1239-4ee9-b7af-f949e9891687\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") "
Nov 02 01:46:18.428965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.428811    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-istio-token\" (UniqueName: \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-istio-token\") pod \"da59dac5-1239-4ee9-b7af-f949e9891687\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") "
Nov 02 01:46:18.428965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.428857    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-data\") pod \"da59dac5-1239-4ee9-b7af-f949e9891687\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") "
Nov 02 01:46:18.428965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.428943    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-certs\") pod \"da59dac5-1239-4ee9-b7af-f949e9891687\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") "
Nov 02 01:46:18.429139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.428982    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-credential-socket\") pod \"da59dac5-1239-4ee9-b7af-f949e9891687\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") "
Nov 02 01:46:18.429139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.429095    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-envoy\") pod \"da59dac5-1239-4ee9-b7af-f949e9891687\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") "
Nov 02 01:46:18.429305 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.429266    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da59dac5-1239-4ee9-b7af-f949e9891687-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da59dac5-1239-4ee9-b7af-f949e9891687-istio-podinfo\") pod \"da59dac5-1239-4ee9-b7af-f949e9891687\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") "
Nov 02 01:46:18.429369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.429323    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da59dac5-1239-4ee9-b7af-f949e9891687-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da59dac5-1239-4ee9-b7af-f949e9891687-istiod-ca-cert\") pod \"da59dac5-1239-4ee9-b7af-f949e9891687\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") "
Nov 02 01:46:18.429369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.429342    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-credential-socket" pod "da59dac5-1239-4ee9-b7af-f949e9891687" (UID: "da59dac5-1239-4ee9-b7af-f949e9891687"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:18.429461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.429419    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-socket\") pod \"da59dac5-1239-4ee9-b7af-f949e9891687\" (UID: \"da59dac5-1239-4ee9-b7af-f949e9891687\") "
Nov 02 01:46:18.430210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.430172    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-data" pod "da59dac5-1239-4ee9-b7af-f949e9891687" (UID: "da59dac5-1239-4ee9-b7af-f949e9891687"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:18.430394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.430253    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:18.430394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.430279    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-certs" pod "da59dac5-1239-4ee9-b7af-f949e9891687" (UID: "da59dac5-1239-4ee9-b7af-f949e9891687"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:18.430541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.430519    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-socket" pod "da59dac5-1239-4ee9-b7af-f949e9891687" (UID: "da59dac5-1239-4ee9-b7af-f949e9891687"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:18.430794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.430754    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da59dac5-1239-4ee9-b7af-f949e9891687-istiod-ca-cert" pod "da59dac5-1239-4ee9-b7af-f949e9891687" (UID: "da59dac5-1239-4ee9-b7af-f949e9891687"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:18.431724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.431689    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-kube-api-access-s6pck" pod "da59dac5-1239-4ee9-b7af-f949e9891687" (UID: "da59dac5-1239-4ee9-b7af-f949e9891687"). InnerVolumeSpecName "kube-api-access-s6pck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:18.459500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.459094    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-istio-token" pod "da59dac5-1239-4ee9-b7af-f949e9891687" (UID: "da59dac5-1239-4ee9-b7af-f949e9891687"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:18.461347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.460539    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-envoy" pod "da59dac5-1239-4ee9-b7af-f949e9891687" (UID: "da59dac5-1239-4ee9-b7af-f949e9891687"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:18.464177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.464148    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da59dac5-1239-4ee9-b7af-f949e9891687-istio-podinfo" pod "da59dac5-1239-4ee9-b7af-f949e9891687" (UID: "da59dac5-1239-4ee9-b7af-f949e9891687"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:46:18.532113 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.532081    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:18.532113 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.532110    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:18.532113 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.532120    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da59dac5-1239-4ee9-b7af-f949e9891687-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:18.532444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.532129    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da59dac5-1239-4ee9-b7af-f949e9891687-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:18.532444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.532138    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:18.532444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.532148    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6pck\" (UniqueName: \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-kube-api-access-s6pck\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:18.532444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.532157    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da59dac5-1239-4ee9-b7af-f949e9891687-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:18.532444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.532168    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da59dac5-1239-4ee9-b7af-f949e9891687-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:18.694999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.694969    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"
Nov 02 01:46:18.699102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.698072    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp" event={"ID":"da59dac5-1239-4ee9-b7af-f949e9891687","Type":"ContainerDied","Data":"16cd0a698ded695787a2270dfa852a51ca3d982e39533d5f49423977246062a5"}
Nov 02 01:46:18.699102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.698148    4851 scope.go:122] "RemoveContainer" containerID="e4e735e6a235d02542919c015c294de6e7f33ceef0038a69157655bf471e4628"
Nov 02 01:46:18.800339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.800308    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"]
Nov 02 01:46:18.804424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.804391    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-rj5dp"]
Nov 02 01:46:18.865047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:18.865010    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" path="/var/lib/kubelet/pods/da59dac5-1239-4ee9-b7af-f949e9891687/volumes"
Nov 02 01:46:19.228564 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:19.228529    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 01:46:19.228810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:19.228579    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 01:46:20.227999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:20.227958    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 01:46:20.228559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:20.228015    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 01:46:21.227759 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:21.227726    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 01:46:21.227969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:21.227790    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 01:46:21.859190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:21.858855    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5" podUID="fe26ff80-ecf0-4541-adc0-56877c9c009d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:46:21.859190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:21.859133    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf" podUID="b3daa532-ae71-4a64-a684-59d4fbb22088" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:46:22.228396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:22.228364    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 01:46:22.228596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:22.228431    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 01:46:23.300472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:23.300205    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"]
Nov 02 01:46:23.315850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:23.315814    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"]
Nov 02 01:46:23.318672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:23.318647    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:46:23.319475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:23.319353    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:46:23.320985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:23.320969    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:46:26.779114 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.779083    4851 generic.go:356] "Generic (PLEG): container finished" podID="fe26ff80-ecf0-4541-adc0-56877c9c009d" containerID="cd55ebf79dbc04e80f5fa0dbff509bc063845c65759293d3ffa8a08e0c65ee71" exitCode=0
Nov 02 01:46:26.779667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.779157    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5" event={"ID":"fe26ff80-ecf0-4541-adc0-56877c9c009d","Type":"ContainerDied","Data":"cd55ebf79dbc04e80f5fa0dbff509bc063845c65759293d3ffa8a08e0c65ee71"}
Nov 02 01:46:26.780990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.780966    4851 generic.go:356] "Generic (PLEG): container finished" podID="b3daa532-ae71-4a64-a684-59d4fbb22088" containerID="6a74d7dcc7cd4f0112f3400d171f6b96d0fa7b6fb9031a72fc9b9e160639b5f5" exitCode=0
Nov 02 01:46:26.781129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.781024    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:46:26.781129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.781043    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf" event={"ID":"b3daa532-ae71-4a64-a684-59d4fbb22088","Type":"ContainerDied","Data":"6a74d7dcc7cd4f0112f3400d171f6b96d0fa7b6fb9031a72fc9b9e160639b5f5"}
Nov 02 01:46:26.781129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.781092    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf" event={"ID":"b3daa532-ae71-4a64-a684-59d4fbb22088","Type":"ContainerDied","Data":"ffd567846d7a1a026e658ebc593be44567ad4540b33e764458e5a5c6e9fd42e7"}
Nov 02 01:46:26.781129 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.781116    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffd567846d7a1a026e658ebc593be44567ad4540b33e764458e5a5c6e9fd42e7"
Nov 02 01:46:26.823728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.823698    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:46:26.829612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.829584    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3daa532-ae71-4a64-a684-59d4fbb22088-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3daa532-ae71-4a64-a684-59d4fbb22088-plugins-config-volume\") pod \"b3daa532-ae71-4a64-a684-59d4fbb22088\" (UID: \"b3daa532-ae71-4a64-a684-59d4fbb22088\") "
Nov 02 01:46:26.829775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.829707    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3daa532-ae71-4a64-a684-59d4fbb22088-kube-api-access-pgvzr\" (UniqueName: \"kubernetes.io/projected/b3daa532-ae71-4a64-a684-59d4fbb22088-kube-api-access-pgvzr\") pod \"b3daa532-ae71-4a64-a684-59d4fbb22088\" (UID: \"b3daa532-ae71-4a64-a684-59d4fbb22088\") "
Nov 02 01:46:26.830252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.830224    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3daa532-ae71-4a64-a684-59d4fbb22088-plugins-config-volume" pod "b3daa532-ae71-4a64-a684-59d4fbb22088" (UID: "b3daa532-ae71-4a64-a684-59d4fbb22088"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:26.832410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.832381    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3daa532-ae71-4a64-a684-59d4fbb22088-kube-api-access-pgvzr" pod "b3daa532-ae71-4a64-a684-59d4fbb22088" (UID: "b3daa532-ae71-4a64-a684-59d4fbb22088"). InnerVolumeSpecName "kube-api-access-pgvzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:26.931156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.931112    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe26ff80-ecf0-4541-adc0-56877c9c009d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe26ff80-ecf0-4541-adc0-56877c9c009d-plugins-config-volume\") pod \"fe26ff80-ecf0-4541-adc0-56877c9c009d\" (UID: \"fe26ff80-ecf0-4541-adc0-56877c9c009d\") "
Nov 02 01:46:26.931156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.931160    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe26ff80-ecf0-4541-adc0-56877c9c009d-kube-api-access-j5pbw\" (UniqueName: \"kubernetes.io/projected/fe26ff80-ecf0-4541-adc0-56877c9c009d-kube-api-access-j5pbw\") pod \"fe26ff80-ecf0-4541-adc0-56877c9c009d\" (UID: \"fe26ff80-ecf0-4541-adc0-56877c9c009d\") "
Nov 02 01:46:26.931643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.931611    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe26ff80-ecf0-4541-adc0-56877c9c009d-plugins-config-volume" pod "fe26ff80-ecf0-4541-adc0-56877c9c009d" (UID: "fe26ff80-ecf0-4541-adc0-56877c9c009d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:26.931860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.931838    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe26ff80-ecf0-4541-adc0-56877c9c009d-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:26.932007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.931985    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3daa532-ae71-4a64-a684-59d4fbb22088-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:26.932092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.932014    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgvzr\" (UniqueName: \"kubernetes.io/projected/b3daa532-ae71-4a64-a684-59d4fbb22088-kube-api-access-pgvzr\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:26.933709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:26.933687    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe26ff80-ecf0-4541-adc0-56877c9c009d-kube-api-access-j5pbw" pod "fe26ff80-ecf0-4541-adc0-56877c9c009d" (UID: "fe26ff80-ecf0-4541-adc0-56877c9c009d"). InnerVolumeSpecName "kube-api-access-j5pbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:27.040446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:27.038893    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pbw\" (UniqueName: \"kubernetes.io/projected/fe26ff80-ecf0-4541-adc0-56877c9c009d-kube-api-access-j5pbw\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:27.794339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:27.794211    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"
Nov 02 01:46:27.794339 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:27.794248    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5" event={"ID":"fe26ff80-ecf0-4541-adc0-56877c9c009d","Type":"ContainerDied","Data":"6abcdeeab0a679c3424b47c81c1efa0b03183c58cf528e7b2b3d441abb9ab8bc"}
Nov 02 01:46:27.794913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:27.794351    4851 scope.go:122] "RemoveContainer" containerID="cd55ebf79dbc04e80f5fa0dbff509bc063845c65759293d3ffa8a08e0c65ee71"
Nov 02 01:46:27.794913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:27.794561    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"
Nov 02 01:46:27.814104 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:27.814050    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"]
Nov 02 01:46:27.819474 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:27.819443    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12e7cb-85757d6779-9ccwf"]
Nov 02 01:46:27.838890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:27.838859    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"]
Nov 02 01:46:27.843644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:27.843609    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-12e7cb-7664f74bbc-njnx5"]
Nov 02 01:46:28.848644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:28.848615    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3daa532-ae71-4a64-a684-59d4fbb22088" path="/var/lib/kubelet/pods/b3daa532-ae71-4a64-a684-59d4fbb22088/volumes"
Nov 02 01:46:28.849059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:28.849045    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe26ff80-ecf0-4541-adc0-56877c9c009d" path="/var/lib/kubelet/pods/fe26ff80-ecf0-4541-adc0-56877c9c009d/volumes"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.121437    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"]
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122363    4851 cpu_manager.go:414] "containerMap: removing container" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122384    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122401    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b3daa532-ae71-4a64-a684-59d4fbb22088" containerName="payload-pre-processing"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122410    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3daa532-ae71-4a64-a684-59d4fbb22088" containerName="payload-pre-processing"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122438    4851 cpu_manager.go:414] "containerMap: removing container" podUID="70565ca1-c120-4b77-b307-a16604644193" containerName="istio-proxy"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122445    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70565ca1-c120-4b77-b307-a16604644193" containerName="istio-proxy"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122499    4851 cpu_manager.go:414] "containerMap: removing container" podUID="fe26ff80-ecf0-4541-adc0-56877c9c009d" containerName="payload-processing"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122505    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe26ff80-ecf0-4541-adc0-56877c9c009d" containerName="payload-processing"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122538    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bb9669fc-e378-4a4b-aea5-ea97c9b71882" containerName="payload-pre-processing"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122544    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9669fc-e378-4a4b-aea5-ea97c9b71882" containerName="payload-pre-processing"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122710    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe26ff80-ecf0-4541-adc0-56877c9c009d" containerName="payload-processing"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122725    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9669fc-e378-4a4b-aea5-ea97c9b71882" containerName="payload-pre-processing"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122736    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122745    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3daa532-ae71-4a64-a684-59d4fbb22088" containerName="payload-pre-processing"
Nov 02 01:46:29.123366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.122756    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="70565ca1-c120-4b77-b307-a16604644193" containerName="istio-proxy"
Nov 02 01:46:29.171391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.171363    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"]
Nov 02 01:46:29.171708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.171650    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.220092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.220055    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:46:29.220262 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.220134    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:46:29.274552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.274519    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm7l9\" (UniqueName: \"kubernetes.io/projected/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-kube-api-access-rm7l9\") pod \"maas-api-revoke-sub-57db3b80-s5fdk\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.274695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.274634    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57db3b80-s5fdk\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.274695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.274687    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57db3b80-s5fdk\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.376042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.375947    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57db3b80-s5fdk\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.376242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.376132    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7l9\" (UniqueName: \"kubernetes.io/projected/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-kube-api-access-rm7l9\") pod \"maas-api-revoke-sub-57db3b80-s5fdk\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.376242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.376229    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57db3b80-s5fdk\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.376891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.376872    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57db3b80-s5fdk\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.378569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.378549    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57db3b80-s5fdk\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.390042 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.390020    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm7l9\" (UniqueName: \"kubernetes.io/projected/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-kube-api-access-rm7l9\") pod \"maas-api-revoke-sub-57db3b80-s5fdk\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.489620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.489279    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:29.782223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.782113    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"]
Nov 02 01:46:29.785636 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:46:29.785608    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b58949e_ba8f_4fd8_8d00_b6b1faeb6b7a.slice/crio-e988254d00e412da6d3ae48a68d61bb7c9c744479e080de8617c8c124d295a12 WatchSource:0}: Error finding container e988254d00e412da6d3ae48a68d61bb7c9c744479e080de8617c8c124d295a12: Status 404 returned error can't find the container with id e988254d00e412da6d3ae48a68d61bb7c9c744479e080de8617c8c124d295a12
Nov 02 01:46:29.822459 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:29.822431    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk" event={"ID":"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a","Type":"ContainerStarted","Data":"e988254d00e412da6d3ae48a68d61bb7c9c744479e080de8617c8c124d295a12"}
Nov 02 01:46:30.833392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:30.833357    4851 generic.go:356] "Generic (PLEG): container finished" podID="1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a" containerID="bfb21a432935dcc62b0372aecf91ba162d79b1ce8c03de17eb3682f40f8b5359" exitCode=0
Nov 02 01:46:30.833830 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:30.833479    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk" event={"ID":"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a","Type":"ContainerDied","Data":"bfb21a432935dcc62b0372aecf91ba162d79b1ce8c03de17eb3682f40f8b5359"}
Nov 02 01:46:31.657747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.657702    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"]
Nov 02 01:46:31.658831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.658733    4851 cpu_manager.go:414] "containerMap: removing container" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy"
Nov 02 01:46:31.658831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.658791    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy"
Nov 02 01:46:31.659047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.658980    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="da59dac5-1239-4ee9-b7af-f949e9891687" containerName="istio-proxy"
Nov 02 01:46:31.663363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.663335    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:31.670919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.670887    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"]
Nov 02 01:46:31.852983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.852947    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8bb21da-lldvk\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:31.853439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.853006    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8bb21da-lldvk\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:31.853439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.853164    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx562\" (UniqueName: \"kubernetes.io/projected/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-kube-api-access-fx562\") pod \"maas-api-revoke-sub-e8bb21da-lldvk\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:31.956584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.955197    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8bb21da-lldvk\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:31.956584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.955249    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8bb21da-lldvk\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:31.956584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.955343    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx562\" (UniqueName: \"kubernetes.io/projected/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-kube-api-access-fx562\") pod \"maas-api-revoke-sub-e8bb21da-lldvk\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:31.956584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.956339    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8bb21da-lldvk\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:31.975075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.974603    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx562\" (UniqueName: \"kubernetes.io/projected/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-kube-api-access-fx562\") pod \"maas-api-revoke-sub-e8bb21da-lldvk\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:31.979319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.978015    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8bb21da-lldvk\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:31.986222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:31.985981    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:32.081311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.078674    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:32.165678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.165641    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-xks-ca\") pod \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") "
Nov 02 01:46:32.166393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.166369    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-openshift-ca\") pod \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") "
Nov 02 01:46:32.167202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.167180    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-kube-api-access-rm7l9\" (UniqueName: \"kubernetes.io/projected/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-kube-api-access-rm7l9\") pod \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\" (UID: \"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a\") "
Nov 02 01:46:32.167730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.166939    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-openshift-ca" pod "1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a" (UID: "1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:32.168469 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.168447    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:32.168885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.168863    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-xks-ca" pod "1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a" (UID: "1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:46:32.182252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.182208    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-kube-api-access-rm7l9" pod "1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a" (UID: "1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a"). InnerVolumeSpecName "kube-api-access-rm7l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:32.269415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.269339    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:32.269415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.269366    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm7l9\" (UniqueName: \"kubernetes.io/projected/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a-kube-api-access-rm7l9\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:32.313242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.313208    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"]
Nov 02 01:46:32.313481 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:46:32.313460    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e2ac7b_34ce_4d14_bf77_479ebc1f35f2.slice/crio-4ae8b7313ca215604e043ecc0878e2fe917b42fe28ed4566cfdcc59331e8e2ab WatchSource:0}: Error finding container 4ae8b7313ca215604e043ecc0878e2fe917b42fe28ed4566cfdcc59331e8e2ab: Status 404 returned error can't find the container with id 4ae8b7313ca215604e043ecc0878e2fe917b42fe28ed4566cfdcc59331e8e2ab
Nov 02 01:46:32.853005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.852928    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"
Nov 02 01:46:32.853005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.852964    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk" event={"ID":"1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a","Type":"ContainerDied","Data":"e988254d00e412da6d3ae48a68d61bb7c9c744479e080de8617c8c124d295a12"}
Nov 02 01:46:32.853522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.853008    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e988254d00e412da6d3ae48a68d61bb7c9c744479e080de8617c8c124d295a12"
Nov 02 01:46:32.861330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.860165    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk" event={"ID":"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2","Type":"ContainerStarted","Data":"7590d0a4fbe1a098893d3be0b47ad22d658e456d30e6d4ad32bf47f53cfece61"}
Nov 02 01:46:32.861330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.860222    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk" event={"ID":"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2","Type":"ContainerStarted","Data":"4ae8b7313ca215604e043ecc0878e2fe917b42fe28ed4566cfdcc59331e8e2ab"}
Nov 02 01:46:32.892929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:32.892878    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk" podStartSLOduration=1.892862429 podStartE2EDuration="1.892862429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:32.886064981 +0000 UTC m=+1898.190997807" watchObservedRunningTime="2026-11-02 01:46:32.892862429 +0000 UTC m=+1898.197795255"
Nov 02 01:46:33.869989 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:33.869950    4851 generic.go:356] "Generic (PLEG): container finished" podID="f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2" containerID="7590d0a4fbe1a098893d3be0b47ad22d658e456d30e6d4ad32bf47f53cfece61" exitCode=0
Nov 02 01:46:33.870431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:33.870035    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk" event={"ID":"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2","Type":"ContainerDied","Data":"7590d0a4fbe1a098893d3be0b47ad22d658e456d30e6d4ad32bf47f53cfece61"}
Nov 02 01:46:35.049264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.049236    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:35.114918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.114883    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-xks-ca\") pod \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") "
Nov 02 01:46:35.115117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.114932    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-openshift-ca\") pod \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") "
Nov 02 01:46:35.115117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.114978    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-kube-api-access-fx562\" (UniqueName: \"kubernetes.io/projected/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-kube-api-access-fx562\") pod \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\" (UID: \"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2\") "
Nov 02 01:46:35.115902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.115856    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-openshift-ca" pod "f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2" (UID: "f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:35.117426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.117403    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-xks-ca" pod "f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2" (UID: "f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:46:35.118881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.118856    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-kube-api-access-fx562" pod "f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2" (UID: "f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2"). InnerVolumeSpecName "kube-api-access-fx562". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:35.217203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.217171    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:35.217203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.217202    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:35.217423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.217213    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx562\" (UniqueName: \"kubernetes.io/projected/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2-kube-api-access-fx562\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:35.887978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.887951    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"
Nov 02 01:46:35.888160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.887999    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk" event={"ID":"f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2","Type":"ContainerDied","Data":"4ae8b7313ca215604e043ecc0878e2fe917b42fe28ed4566cfdcc59331e8e2ab"}
Nov 02 01:46:35.888160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:35.888047    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae8b7313ca215604e043ecc0878e2fe917b42fe28ed4566cfdcc59331e8e2ab"
Nov 02 01:46:38.330866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.330830    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main" containerID="cri-o://22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b" gracePeriod=45
Nov 02 01:46:38.345421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.345381    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main" containerID="cri-o://221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7" gracePeriod=45
Nov 02 01:46:38.628204 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.628163    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:46:38.653111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.653085    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:46:38.687340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.687306    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-dshm\") pod \"6781693f-2ff0-449d-b191-ec010c6cb045\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") "
Nov 02 01:46:38.687537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.687426    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-model-cache\") pod \"6781693f-2ff0-449d-b191-ec010c6cb045\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") "
Nov 02 01:46:38.687537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.687469    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-tmp-dir\") pod \"6781693f-2ff0-449d-b191-ec010c6cb045\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") "
Nov 02 01:46:38.687537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.687495    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-home\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-home\") pod \"6781693f-2ff0-449d-b191-ec010c6cb045\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") "
Nov 02 01:46:38.687721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.687596    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6781693f-2ff0-449d-b191-ec010c6cb045-kube-api-access-5l46z\" (UniqueName: \"kubernetes.io/projected/6781693f-2ff0-449d-b191-ec010c6cb045-kube-api-access-5l46z\") pod \"6781693f-2ff0-449d-b191-ec010c6cb045\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") "
Nov 02 01:46:38.687721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.687677    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6781693f-2ff0-449d-b191-ec010c6cb045-tls-certs\" (UniqueName: \"kubernetes.io/secret/6781693f-2ff0-449d-b191-ec010c6cb045-tls-certs\") pod \"6781693f-2ff0-449d-b191-ec010c6cb045\" (UID: \"6781693f-2ff0-449d-b191-ec010c6cb045\") "
Nov 02 01:46:38.689683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.689639    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-home" pod "6781693f-2ff0-449d-b191-ec010c6cb045" (UID: "6781693f-2ff0-449d-b191-ec010c6cb045"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:38.692243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.692208    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6781693f-2ff0-449d-b191-ec010c6cb045-tls-certs" pod "6781693f-2ff0-449d-b191-ec010c6cb045" (UID: "6781693f-2ff0-449d-b191-ec010c6cb045"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:46:38.692713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.692675    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-tmp-dir" pod "6781693f-2ff0-449d-b191-ec010c6cb045" (UID: "6781693f-2ff0-449d-b191-ec010c6cb045"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:38.692891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.692854    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-dshm" pod "6781693f-2ff0-449d-b191-ec010c6cb045" (UID: "6781693f-2ff0-449d-b191-ec010c6cb045"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:38.693039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.693008    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-model-cache" pod "6781693f-2ff0-449d-b191-ec010c6cb045" (UID: "6781693f-2ff0-449d-b191-ec010c6cb045"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:38.698595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.698125    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6781693f-2ff0-449d-b191-ec010c6cb045-kube-api-access-5l46z" pod "6781693f-2ff0-449d-b191-ec010c6cb045" (UID: "6781693f-2ff0-449d-b191-ec010c6cb045"). InnerVolumeSpecName "kube-api-access-5l46z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:38.789680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.789651    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-model-cache\") pod \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") "
Nov 02 01:46:38.789879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.789698    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tmp-dir\") pod \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") "
Nov 02 01:46:38.789879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.789731    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-dshm\") pod \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") "
Nov 02 01:46:38.789879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.789757    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tls-certs\") pod \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") "
Nov 02 01:46:38.789879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.789862    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-kube-api-access-v5rgp\" (UniqueName: \"kubernetes.io/projected/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-kube-api-access-v5rgp\") pod \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") "
Nov 02 01:46:38.790143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.789895    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-home\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-home\") pod \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\" (UID: \"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d\") "
Nov 02 01:46:38.790143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.789965    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-model-cache" pod "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" (UID: "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:38.790143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790085    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tmp-dir" pod "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" (UID: "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:38.790323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790194    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-home" pod "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" (UID: "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:38.795380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790581    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.795380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790606    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.795380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790622    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.795380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790635    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.795380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790647    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.795380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790658    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6781693f-2ff0-449d-b191-ec010c6cb045-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.795380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790673    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l46z\" (UniqueName: \"kubernetes.io/projected/6781693f-2ff0-449d-b191-ec010c6cb045-kube-api-access-5l46z\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.795380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790686    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6781693f-2ff0-449d-b191-ec010c6cb045-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.795380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.790697    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.799900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.797161    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-dshm" pod "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" (UID: "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:38.799900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.799707    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tls-certs" pod "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" (UID: "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:46:38.800738 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.800714    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-kube-api-access-v5rgp" pod "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" (UID: "d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d"). InnerVolumeSpecName "kube-api-access-v5rgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:38.892620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.892583    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5rgp\" (UniqueName: \"kubernetes.io/projected/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-kube-api-access-v5rgp\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.892620 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.892621    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.892829 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.892635    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:38.940416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.940355    4851 generic.go:356] "Generic (PLEG): container finished" podID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerID="221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7" exitCode=0
Nov 02 01:46:38.941063 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.941017    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"
Nov 02 01:46:38.942633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.942588    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" event={"ID":"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d","Type":"ContainerDied","Data":"221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7"}
Nov 02 01:46:38.942825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.942806    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph" event={"ID":"d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d","Type":"ContainerDied","Data":"2eb38df7f3935a6c3c2eba274dc8e494f082df91f7557a7ba1e9d5070b960e04"}
Nov 02 01:46:38.942940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.942928    4851 scope.go:122] "RemoveContainer" containerID="221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7"
Nov 02 01:46:38.983031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.983001    4851 generic.go:356] "Generic (PLEG): container finished" podID="6781693f-2ff0-449d-b191-ec010c6cb045" containerID="22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b" exitCode=0
Nov 02 01:46:38.983336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.983316    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" event={"ID":"6781693f-2ff0-449d-b191-ec010c6cb045","Type":"ContainerDied","Data":"22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b"}
Nov 02 01:46:38.983459 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.983446    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g" event={"ID":"6781693f-2ff0-449d-b191-ec010c6cb045","Type":"ContainerDied","Data":"c80933d4abb9fc15b0d02b5884588a26b4013501e84c9981477abb9130256e69"}
Nov 02 01:46:38.983625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.983612    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"
Nov 02 01:46:38.996603 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:38.996562    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"]
Nov 02 01:46:39.001701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:39.001675    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/premium-w6-b9d816-kserve-765b7b6db-bppph"]
Nov 02 01:46:39.004165 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:39.004143    4851 scope.go:122] "RemoveContainer" containerID="221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7"
Nov 02 01:46:39.004483 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:46:39.004462    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7\": container with ID starting with 221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7 not found: ID does not exist" containerID="221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7"
Nov 02 01:46:39.004550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:39.004492    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7"} err="failed to get container status \"221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7\": rpc error: code = NotFound desc = could not find container \"221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7\": container with ID starting with 221583a40e9dcf098872ee2f0305efae50447cdc10f7c4d2f9567dd511cb45d7 not found: ID does not exist"
Nov 02 01:46:39.004550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:39.004512    4851 scope.go:122] "RemoveContainer" containerID="22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b"
Nov 02 01:46:39.012850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:39.012821    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"]
Nov 02 01:46:39.016001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:39.015983    4851 scope.go:122] "RemoveContainer" containerID="22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b"
Nov 02 01:46:39.016325 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:46:39.016305    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b\": container with ID starting with 22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b not found: ID does not exist" containerID="22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b"
Nov 02 01:46:39.016382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:39.016330    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b"} err="failed to get container status \"22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b\": rpc error: code = NotFound desc = could not find container \"22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b\": container with ID starting with 22c1db60095d92fd61ccf45549013644f898bf52e6f56a70d256208fa3bef83b not found: ID does not exist"
Nov 02 01:46:39.016518 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:39.016501    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/distinct-2-w6-b9d816-kserve-d49c695f4-8ss9g"]
Nov 02 01:46:40.848133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:40.848101    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" path="/var/lib/kubelet/pods/6781693f-2ff0-449d-b191-ec010c6cb045/volumes"
Nov 02 01:46:40.848630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:40.848615    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" path="/var/lib/kubelet/pods/d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d/volumes"
Nov 02 01:46:47.661501 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:46:47.661466    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:46:49.508051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:49.507108    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"]
Nov 02 01:46:49.508051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:49.507359    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy" containerID="cri-o://28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5" gracePeriod=30
Nov 02 01:46:53.099590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:53.099545    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"]
Nov 02 01:46:53.100073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:53.099832    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy" containerID="cri-o://6ee873fff7889adb11a174b3521c5331556dfa29f04f1c0b3eecf344d9bce178" gracePeriod=30
Nov 02 01:46:53.105545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:53.105518    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"]
Nov 02 01:46:53.105730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:53.105697    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" containerID="cri-o://aa118a256d9aaf6d1f450aa0b776ea3efa3c0772fc98dbb423dce4dac99a6bee" gracePeriod=30
Nov 02 01:46:53.879901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:53.879865    4851 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:46:53.880090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:53.879919    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:46:54.896317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.896275    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:46:54.967818 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.967772    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8e45f01-7368-4448-bbec-2557dd42e004-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e45f01-7368-4448-bbec-2557dd42e004-istio-podinfo\") pod \"f8e45f01-7368-4448-bbec-2557dd42e004\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") "
Nov 02 01:46:54.967997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.967860    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-certs\") pod \"f8e45f01-7368-4448-bbec-2557dd42e004\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") "
Nov 02 01:46:54.967997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.967907    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-credential-socket\") pod \"f8e45f01-7368-4448-bbec-2557dd42e004\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") "
Nov 02 01:46:54.968147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.968006    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-istio-token\" (UniqueName: \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-istio-token\") pod \"f8e45f01-7368-4448-bbec-2557dd42e004\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") "
Nov 02 01:46:54.968147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.968067    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-data\") pod \"f8e45f01-7368-4448-bbec-2557dd42e004\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") "
Nov 02 01:46:54.968147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.968115    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-socket\") pod \"f8e45f01-7368-4448-bbec-2557dd42e004\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") "
Nov 02 01:46:54.968147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.968142    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-envoy\") pod \"f8e45f01-7368-4448-bbec-2557dd42e004\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") "
Nov 02 01:46:54.968406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.968215    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e45f01-7368-4448-bbec-2557dd42e004-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e45f01-7368-4448-bbec-2557dd42e004-istiod-ca-cert\") pod \"f8e45f01-7368-4448-bbec-2557dd42e004\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") "
Nov 02 01:46:54.968406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.968249    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-kube-api-access-bqvbq\" (UniqueName: \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-kube-api-access-bqvbq\") pod \"f8e45f01-7368-4448-bbec-2557dd42e004\" (UID: \"f8e45f01-7368-4448-bbec-2557dd42e004\") "
Nov 02 01:46:54.968945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.968817    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-certs" pod "f8e45f01-7368-4448-bbec-2557dd42e004" (UID: "f8e45f01-7368-4448-bbec-2557dd42e004"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:54.969143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.969092    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:54.969143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.969105    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-socket" pod "f8e45f01-7368-4448-bbec-2557dd42e004" (UID: "f8e45f01-7368-4448-bbec-2557dd42e004"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:54.970599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.970283    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e45f01-7368-4448-bbec-2557dd42e004-istiod-ca-cert" pod "f8e45f01-7368-4448-bbec-2557dd42e004" (UID: "f8e45f01-7368-4448-bbec-2557dd42e004"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:54.971198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.971178    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-data" pod "f8e45f01-7368-4448-bbec-2557dd42e004" (UID: "f8e45f01-7368-4448-bbec-2557dd42e004"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:54.971563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.971544    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-credential-socket" pod "f8e45f01-7368-4448-bbec-2557dd42e004" (UID: "f8e45f01-7368-4448-bbec-2557dd42e004"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:54.985367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.973477    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-envoy" pod "f8e45f01-7368-4448-bbec-2557dd42e004" (UID: "f8e45f01-7368-4448-bbec-2557dd42e004"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:54.985367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.978029    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-istio-token" pod "f8e45f01-7368-4448-bbec-2557dd42e004" (UID: "f8e45f01-7368-4448-bbec-2557dd42e004"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:54.985367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.978722    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:46:54.985367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.978797    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:46:54.985367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.981402    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-kube-api-access-bqvbq" pod "f8e45f01-7368-4448-bbec-2557dd42e004" (UID: "f8e45f01-7368-4448-bbec-2557dd42e004"). InnerVolumeSpecName "kube-api-access-bqvbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:54.985367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:54.984000    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8e45f01-7368-4448-bbec-2557dd42e004-istio-podinfo" pod "f8e45f01-7368-4448-bbec-2557dd42e004" (UID: "f8e45f01-7368-4448-bbec-2557dd42e004"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:46:55.079389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.077616    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:55.079389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.077654    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:55.079389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.077669    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:55.079389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.077683    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e45f01-7368-4448-bbec-2557dd42e004-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:55.079389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.077698    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqvbq\" (UniqueName: \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-kube-api-access-bqvbq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:55.079389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.077714    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e45f01-7368-4448-bbec-2557dd42e004-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:55.079389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.077729    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e45f01-7368-4448-bbec-2557dd42e004-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:55.079389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.077744    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e45f01-7368-4448-bbec-2557dd42e004-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:55.184153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.184100    4851 generic.go:356] "Generic (PLEG): container finished" podID="f8e45f01-7368-4448-bbec-2557dd42e004" containerID="28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5" exitCode=0
Nov 02 01:46:55.185458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.184525    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"
Nov 02 01:46:55.187690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.187358    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" event={"ID":"f8e45f01-7368-4448-bbec-2557dd42e004","Type":"ContainerDied","Data":"28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5"}
Nov 02 01:46:55.187690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.187451    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc" event={"ID":"f8e45f01-7368-4448-bbec-2557dd42e004","Type":"ContainerDied","Data":"6da843f7868e0e19c76b2cbfb855f7bd2a741bcc83c2609314199808576f093b"}
Nov 02 01:46:55.187690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.187480    4851 scope.go:122] "RemoveContainer" containerID="28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5"
Nov 02 01:46:55.252266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.251541    4851 scope.go:122] "RemoveContainer" containerID="2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5"
Nov 02 01:46:55.266259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.265911    4851 scope.go:122] "RemoveContainer" containerID="28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5"
Nov 02 01:46:55.266450 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:46:55.266423    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5\": container with ID starting with 28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5 not found: ID does not exist" containerID="28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5"
Nov 02 01:46:55.266533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.266466    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5"} err="failed to get container status \"28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5\": rpc error: code = NotFound desc = could not find container \"28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5\": container with ID starting with 28995cd24a53cb5851635c7f0de92f88abdbd0c54ef102737a6ad54041b3f4d5 not found: ID does not exist"
Nov 02 01:46:55.266533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.266490    4851 scope.go:122] "RemoveContainer" containerID="2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5"
Nov 02 01:46:55.266813 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:46:55.266755    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5\": container with ID starting with 2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5 not found: ID does not exist" containerID="2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5"
Nov 02 01:46:55.266885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.266819    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5"} err="failed to get container status \"2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5\": rpc error: code = NotFound desc = could not find container \"2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5\": container with ID starting with 2fcee0c0aa9bec844803644ef9e971564d9fc9652dad3fb368b433606ef66af5 not found: ID does not exist"
Nov 02 01:46:55.270389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.270370    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"]
Nov 02 01:46:55.280119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:55.280095    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-2mxbc"]
Nov 02 01:46:56.760203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:56.760162    4851 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:46:56.760618 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:56.760228    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:46:56.848953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:56.848918    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" path="/var/lib/kubelet/pods/f8e45f01-7368-4448-bbec-2557dd42e004/volumes"
Nov 02 01:46:58.231368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.231333    4851 generic.go:356] "Generic (PLEG): container finished" podID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerID="aa118a256d9aaf6d1f450aa0b776ea3efa3c0772fc98dbb423dce4dac99a6bee" exitCode=0
Nov 02 01:46:58.231901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.231410    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" event={"ID":"afc5860f-cfc3-4caf-844b-92dcbe2d91e4","Type":"ContainerDied","Data":"aa118a256d9aaf6d1f450aa0b776ea3efa3c0772fc98dbb423dce4dac99a6bee"}
Nov 02 01:46:58.235069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.234668    4851 generic.go:356] "Generic (PLEG): container finished" podID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerID="6ee873fff7889adb11a174b3521c5331556dfa29f04f1c0b3eecf344d9bce178" exitCode=0
Nov 02 01:46:58.235069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.234882    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" event={"ID":"a3849006-d06b-4c97-840d-5f34aa6cdfe1","Type":"ContainerDied","Data":"6ee873fff7889adb11a174b3521c5331556dfa29f04f1c0b3eecf344d9bce178"}
Nov 02 01:46:58.235069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.234934    4851 scope.go:122] "RemoveContainer" containerID="31b2bdd4e82af455230be053b83d6fbaed4cb1d93516a41beb780de730fcbeba"
Nov 02 01:46:58.408252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.408222    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:46:58.420765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.420734    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istiod-ca-cert\") pod \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") "
Nov 02 01:46:58.420983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.420789    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-credential-socket\") pod \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") "
Nov 02 01:46:58.420983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.420819    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-envoy\") pod \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") "
Nov 02 01:46:58.420983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.420844    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-data\") pod \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") "
Nov 02 01:46:58.420983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.420889    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-podinfo\") pod \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") "
Nov 02 01:46:58.420983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.420970    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-kube-api-access-pxh6w\" (UniqueName: \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-kube-api-access-pxh6w\") pod \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") "
Nov 02 01:46:58.421270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.420999    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-socket\") pod \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") "
Nov 02 01:46:58.421270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421179    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-token\" (UniqueName: \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-token\") pod \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") "
Nov 02 01:46:58.421270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421192    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istiod-ca-cert" pod "a3849006-d06b-4c97-840d-5f34aa6cdfe1" (UID: "a3849006-d06b-4c97-840d-5f34aa6cdfe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:58.421270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421199    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-credential-socket" pod "a3849006-d06b-4c97-840d-5f34aa6cdfe1" (UID: "a3849006-d06b-4c97-840d-5f34aa6cdfe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:58.421270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421215    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-certs\") pod \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\" (UID: \"a3849006-d06b-4c97-840d-5f34aa6cdfe1\") "
Nov 02 01:46:58.421638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421484    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-socket" pod "a3849006-d06b-4c97-840d-5f34aa6cdfe1" (UID: "a3849006-d06b-4c97-840d-5f34aa6cdfe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:58.421638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421513    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-data" pod "a3849006-d06b-4c97-840d-5f34aa6cdfe1" (UID: "a3849006-d06b-4c97-840d-5f34aa6cdfe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:58.421761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421743    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-certs" pod "a3849006-d06b-4c97-840d-5f34aa6cdfe1" (UID: "a3849006-d06b-4c97-840d-5f34aa6cdfe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:58.421838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421807    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.421838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421827    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.421962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421842    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.421962 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.421855    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.424153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.424125    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-token" pod "a3849006-d06b-4c97-840d-5f34aa6cdfe1" (UID: "a3849006-d06b-4c97-840d-5f34aa6cdfe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:58.424587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.424255    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-podinfo" pod "a3849006-d06b-4c97-840d-5f34aa6cdfe1" (UID: "a3849006-d06b-4c97-840d-5f34aa6cdfe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:46:58.424587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.424326    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-kube-api-access-pxh6w" pod "a3849006-d06b-4c97-840d-5f34aa6cdfe1" (UID: "a3849006-d06b-4c97-840d-5f34aa6cdfe1"). InnerVolumeSpecName "kube-api-access-pxh6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:58.424587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.424403    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-envoy" pod "a3849006-d06b-4c97-840d-5f34aa6cdfe1" (UID: "a3849006-d06b-4c97-840d-5f34aa6cdfe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:58.449110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.449080    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.523675    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-data\") pod \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") "
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.523784    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istiod-ca-cert\") pod \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") "
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.523828    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-envoy\") pod \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") "
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.523890    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-token\" (UniqueName: \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-token\") pod \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") "
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.523922    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-credential-socket\") pod \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") "
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.523975    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-kube-api-access-6hpr9\" (UniqueName: \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-kube-api-access-6hpr9\") pod \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") "
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524008    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-data" pod "afc5860f-cfc3-4caf-844b-92dcbe2d91e4" (UID: "afc5860f-cfc3-4caf-844b-92dcbe2d91e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524086    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-socket\") pod \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") "
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524105    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-podinfo\") pod \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") "
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524155    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-certs\") pod \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\" (UID: \"afc5860f-cfc3-4caf-844b-92dcbe2d91e4\") "
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524409    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-credential-socket" pod "afc5860f-cfc3-4caf-844b-92dcbe2d91e4" (UID: "afc5860f-cfc3-4caf-844b-92dcbe2d91e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524421    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istiod-ca-cert" pod "afc5860f-cfc3-4caf-844b-92dcbe2d91e4" (UID: "afc5860f-cfc3-4caf-844b-92dcbe2d91e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524654    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-socket" pod "afc5860f-cfc3-4caf-844b-92dcbe2d91e4" (UID: "afc5860f-cfc3-4caf-844b-92dcbe2d91e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524722    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-certs" pod "afc5860f-cfc3-4caf-844b-92dcbe2d91e4" (UID: "afc5860f-cfc3-4caf-844b-92dcbe2d91e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524853    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524871    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524885    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524899    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524912    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524927    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524942    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3849006-d06b-4c97-840d-5f34aa6cdfe1-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524955    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.524970    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxh6w\" (UniqueName: \"kubernetes.io/projected/a3849006-d06b-4c97-840d-5f34aa6cdfe1-kube-api-access-pxh6w\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.525013    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.528401    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-podinfo" pod "afc5860f-cfc3-4caf-844b-92dcbe2d91e4" (UID: "afc5860f-cfc3-4caf-844b-92dcbe2d91e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.528423    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-kube-api-access-6hpr9" pod "afc5860f-cfc3-4caf-844b-92dcbe2d91e4" (UID: "afc5860f-cfc3-4caf-844b-92dcbe2d91e4"). InnerVolumeSpecName "kube-api-access-6hpr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.528562    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-token" pod "afc5860f-cfc3-4caf-844b-92dcbe2d91e4" (UID: "afc5860f-cfc3-4caf-844b-92dcbe2d91e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:46:58.535316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.533959    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-envoy" pod "afc5860f-cfc3-4caf-844b-92dcbe2d91e4" (UID: "afc5860f-cfc3-4caf-844b-92dcbe2d91e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:46:58.626044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.626011    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.626044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.626038    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.626044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.626047    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:58.626307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:58.626056    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hpr9\" (UniqueName: \"kubernetes.io/projected/afc5860f-cfc3-4caf-844b-92dcbe2d91e4-kube-api-access-6hpr9\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:46:59.244268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:59.244238    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"
Nov 02 01:46:59.244747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:59.244258    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f" event={"ID":"a3849006-d06b-4c97-840d-5f34aa6cdfe1","Type":"ContainerDied","Data":"350cd280e339afb35935557c5bfe264e59d477d3aad50dfba04258a510eab79b"}
Nov 02 01:46:59.244747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:59.244348    4851 scope.go:122] "RemoveContainer" containerID="6ee873fff7889adb11a174b3521c5331556dfa29f04f1c0b3eecf344d9bce178"
Nov 02 01:46:59.246251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:59.246233    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"
Nov 02 01:46:59.246251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:59.246236    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv" event={"ID":"afc5860f-cfc3-4caf-844b-92dcbe2d91e4","Type":"ContainerDied","Data":"0b77e6946154d2a8e5e8d36202f42428126704cd325c3784031f3ae22a2c8010"}
Nov 02 01:46:59.257273 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:59.257245    4851 scope.go:122] "RemoveContainer" containerID="aa118a256d9aaf6d1f450aa0b776ea3efa3c0772fc98dbb423dce4dac99a6bee"
Nov 02 01:46:59.265563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:59.265501    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"]
Nov 02 01:46:59.271511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:59.271491    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-8jd2f"]
Nov 02 01:46:59.286101 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:59.286057    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"]
Nov 02 01:46:59.295565 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:46:59.295539    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-mtsmv"]
Nov 02 01:47:00.848662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:00.848625    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" path="/var/lib/kubelet/pods/a3849006-d06b-4c97-840d-5f34aa6cdfe1/volumes"
Nov 02 01:47:00.849441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:00.849421    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" path="/var/lib/kubelet/pods/afc5860f-cfc3-4caf-844b-92dcbe2d91e4/volumes"
Nov 02 01:47:02.470337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:02.470274    4851 scope.go:122] "RemoveContainer" containerID="6a74d7dcc7cd4f0112f3400d171f6b96d0fa7b6fb9031a72fc9b9e160639b5f5"
Nov 02 01:47:17.683173 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:47:17.683092    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:47:37.025086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:37.025039    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"]
Nov 02 01:47:37.029221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:37.029186    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1907de4a-2jxfw"]
Nov 02 01:47:38.023229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:38.023149    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"]
Nov 02 01:47:38.034995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:38.034955    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c525c346-sh8k2"]
Nov 02 01:47:38.847852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:38.847818    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74674aa0-450d-403d-8cde-95e91a71e7ab" path="/var/lib/kubelet/pods/74674aa0-450d-403d-8cde-95e91a71e7ab/volumes"
Nov 02 01:47:38.848348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:38.848333    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908a6786-06a1-4323-9157-3de0777f0627" path="/var/lib/kubelet/pods/908a6786-06a1-4323-9157-3de0777f0627/volumes"
Nov 02 01:47:47.657256 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:47:47.657211    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:47:49.024253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:49.024217    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"]
Nov 02 01:47:49.027350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:49.027327    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54f57396-vwfn5"]
Nov 02 01:47:50.024123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:50.024088    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"]
Nov 02 01:47:50.027038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:50.027015    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f47a381-6kljw"]
Nov 02 01:47:50.886686 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:50.886634    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6650af1a-9d41-48df-9df0-dafa1d85a2cc" path="/var/lib/kubelet/pods/6650af1a-9d41-48df-9df0-dafa1d85a2cc/volumes"
Nov 02 01:47:50.887593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:47:50.887570    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6460e6-9466-45fe-a13c-22a1ae10e912" path="/var/lib/kubelet/pods/bb6460e6-9466-45fe-a13c-22a1ae10e912/volumes"
Nov 02 01:48:02.025640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:02.025605    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"]
Nov 02 01:48:02.028677 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:02.028653    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-befba964-wgl5g"]
Nov 02 01:48:02.847491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:02.847461    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338501e4-000f-4aed-92d7-97cae792d292" path="/var/lib/kubelet/pods/338501e4-000f-4aed-92d7-97cae792d292/volumes"
Nov 02 01:48:07.535381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535343    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"]
Nov 02 01:48:07.535912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535898    4851 cpu_manager.go:414] "containerMap: removing container" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy"
Nov 02 01:48:07.535968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535915    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy"
Nov 02 01:48:07.535968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535931    4851 cpu_manager.go:414] "containerMap: removing container" podUID="1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a" containerName="revoke-keys"
Nov 02 01:48:07.535968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535937    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a" containerName="revoke-keys"
Nov 02 01:48:07.535968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535950    4851 cpu_manager.go:414] "containerMap: removing container" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy"
Nov 02 01:48:07.535968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535954    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy"
Nov 02 01:48:07.535968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535964    4851 cpu_manager.go:414] "containerMap: removing container" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy"
Nov 02 01:48:07.535968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535969    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535981    4851 cpu_manager.go:414] "containerMap: removing container" podUID="f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2" containerName="revoke-keys"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.535986    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2" containerName="revoke-keys"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536001    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536005    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536012    4851 cpu_manager.go:414] "containerMap: removing container" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536017    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536025    4851 cpu_manager.go:414] "containerMap: removing container" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536029    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536043    4851 cpu_manager.go:414] "containerMap: removing container" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536047    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536134    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b8ea2a-5325-4ed7-8988-2c46ec6eda1d" containerName="main"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536142    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="6781693f-2ff0-449d-b191-ec010c6cb045" containerName="main"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536150    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc5860f-cfc3-4caf-844b-92dcbe2d91e4" containerName="istio-proxy"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536156    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536164    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536169    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a" containerName="revoke-keys"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536174    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3849006-d06b-4c97-840d-5f34aa6cdfe1" containerName="istio-proxy"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536180    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e45f01-7368-4448-bbec-2557dd42e004" containerName="istio-proxy"
Nov 02 01:48:07.536195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.536186    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2" containerName="revoke-keys"
Nov 02 01:48:07.541360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.541338    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:07.545572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.545548    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:48:07.547914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.547888    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"]
Nov 02 01:48:07.646439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.646403    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:07.648665 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.648561    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t4gg\" (UniqueName: \"kubernetes.io/projected/5de2b849-a2a8-49fb-9a72-691a857f676a-kube-api-access-9t4gg\") pod \"maas-api-revoke-sub-d6e430dd-w7nd6\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:07.649134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.649107    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e430dd-w7nd6\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:07.649853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.649826    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e430dd-w7nd6\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:07.751168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.751132    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t4gg\" (UniqueName: \"kubernetes.io/projected/5de2b849-a2a8-49fb-9a72-691a857f676a-kube-api-access-9t4gg\") pod \"maas-api-revoke-sub-d6e430dd-w7nd6\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:07.751414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.751188    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e430dd-w7nd6\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:07.751414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.751281    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e430dd-w7nd6\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:07.751871 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.751842    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e430dd-w7nd6\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:07.753874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.753856    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e430dd-w7nd6\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:07.763656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.763631    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t4gg\" (UniqueName: \"kubernetes.io/projected/5de2b849-a2a8-49fb-9a72-691a857f676a-kube-api-access-9t4gg\") pod \"maas-api-revoke-sub-d6e430dd-w7nd6\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:07.858880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:07.858801    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:08.150117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:08.150085    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"]
Nov 02 01:48:08.152348 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:48:08.152317    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de2b849_a2a8_49fb_9a72_691a857f676a.slice/crio-f7bebd67b56e140d6649911eb21be0ac09bf0462e8ab26068294a1c20d9130ae WatchSource:0}: Error finding container f7bebd67b56e140d6649911eb21be0ac09bf0462e8ab26068294a1c20d9130ae: Status 404 returned error can't find the container with id f7bebd67b56e140d6649911eb21be0ac09bf0462e8ab26068294a1c20d9130ae
Nov 02 01:48:08.923650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:08.923618    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6" event={"ID":"5de2b849-a2a8-49fb-9a72-691a857f676a","Type":"ContainerStarted","Data":"1a691f5b8f3b8b1fe708bf3871177035dfae3d6ef28fcbac686d54eda66797b3"}
Nov 02 01:48:08.924239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:08.923674    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6" event={"ID":"5de2b849-a2a8-49fb-9a72-691a857f676a","Type":"ContainerStarted","Data":"f7bebd67b56e140d6649911eb21be0ac09bf0462e8ab26068294a1c20d9130ae"}
Nov 02 01:48:08.946587 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:08.946541    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6" podStartSLOduration=1.946527554 podStartE2EDuration="1.946527554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:08.942318224 +0000 UTC m=+1994.247251051" watchObservedRunningTime="2026-11-02 01:48:08.946527554 +0000 UTC m=+1994.251460379"
Nov 02 01:48:09.959803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:09.959764    4851 generic.go:356] "Generic (PLEG): container finished" podID="5de2b849-a2a8-49fb-9a72-691a857f676a" containerID="1a691f5b8f3b8b1fe708bf3871177035dfae3d6ef28fcbac686d54eda66797b3" exitCode=0
Nov 02 01:48:09.960350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:09.959861    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6" event={"ID":"5de2b849-a2a8-49fb-9a72-691a857f676a","Type":"ContainerDied","Data":"1a691f5b8f3b8b1fe708bf3871177035dfae3d6ef28fcbac686d54eda66797b3"}
Nov 02 01:48:11.219650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.219568    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:11.313851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.313815    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-openshift-ca\") pod \"5de2b849-a2a8-49fb-9a72-691a857f676a\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") "
Nov 02 01:48:11.314022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.313948    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de2b849-a2a8-49fb-9a72-691a857f676a-kube-api-access-9t4gg\" (UniqueName: \"kubernetes.io/projected/5de2b849-a2a8-49fb-9a72-691a857f676a-kube-api-access-9t4gg\") pod \"5de2b849-a2a8-49fb-9a72-691a857f676a\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") "
Nov 02 01:48:11.314022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.313976    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-xks-ca\") pod \"5de2b849-a2a8-49fb-9a72-691a857f676a\" (UID: \"5de2b849-a2a8-49fb-9a72-691a857f676a\") "
Nov 02 01:48:11.314200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.314174    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-openshift-ca" pod "5de2b849-a2a8-49fb-9a72-691a857f676a" (UID: "5de2b849-a2a8-49fb-9a72-691a857f676a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:11.314490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.314471    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:11.316215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.316182    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-xks-ca" pod "5de2b849-a2a8-49fb-9a72-691a857f676a" (UID: "5de2b849-a2a8-49fb-9a72-691a857f676a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:11.316395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.316273    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de2b849-a2a8-49fb-9a72-691a857f676a-kube-api-access-9t4gg" pod "5de2b849-a2a8-49fb-9a72-691a857f676a" (UID: "5de2b849-a2a8-49fb-9a72-691a857f676a"). InnerVolumeSpecName "kube-api-access-9t4gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:11.415342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.415279    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t4gg\" (UniqueName: \"kubernetes.io/projected/5de2b849-a2a8-49fb-9a72-691a857f676a-kube-api-access-9t4gg\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:11.415342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.415335    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de2b849-a2a8-49fb-9a72-691a857f676a-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:11.983261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.983228    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"
Nov 02 01:48:11.983522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.983324    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6" event={"ID":"5de2b849-a2a8-49fb-9a72-691a857f676a","Type":"ContainerDied","Data":"f7bebd67b56e140d6649911eb21be0ac09bf0462e8ab26068294a1c20d9130ae"}
Nov 02 01:48:11.983522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:11.983366    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7bebd67b56e140d6649911eb21be0ac09bf0462e8ab26068294a1c20d9130ae"
Nov 02 01:48:17.656851 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:48:17.656811    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:48:17.742520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.742485    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"]
Nov 02 01:48:17.743073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.743059    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5de2b849-a2a8-49fb-9a72-691a857f676a" containerName="revoke-keys"
Nov 02 01:48:17.743119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.743074    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de2b849-a2a8-49fb-9a72-691a857f676a" containerName="revoke-keys"
Nov 02 01:48:17.743170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.743161    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de2b849-a2a8-49fb-9a72-691a857f676a" containerName="revoke-keys"
Nov 02 01:48:17.746614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.746595    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:17.752481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.752453    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:17.752609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.752566    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:48:17.757147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.756975    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"]
Nov 02 01:48:17.911545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.911519    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnkp9\" (UniqueName: \"kubernetes.io/projected/999c26e2-095c-4233-8206-147d729428cd-kube-api-access-fnkp9\") pod \"maas-api-revoke-sub-4e181c48-wr8jt\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:17.911705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.911646    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c26e2-095c-4233-8206-147d729428cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e181c48-wr8jt\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:17.911751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:17.911712    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c26e2-095c-4233-8206-147d729428cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e181c48-wr8jt\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:18.016622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:18.015264    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnkp9\" (UniqueName: \"kubernetes.io/projected/999c26e2-095c-4233-8206-147d729428cd-kube-api-access-fnkp9\") pod \"maas-api-revoke-sub-4e181c48-wr8jt\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:18.016622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:18.015482    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c26e2-095c-4233-8206-147d729428cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e181c48-wr8jt\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:18.016622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:18.015581    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c26e2-095c-4233-8206-147d729428cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e181c48-wr8jt\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:18.027226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:18.027191    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c26e2-095c-4233-8206-147d729428cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e181c48-wr8jt\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:18.031258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:18.031230    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c26e2-095c-4233-8206-147d729428cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e181c48-wr8jt\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:18.031509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:18.031486    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnkp9\" (UniqueName: \"kubernetes.io/projected/999c26e2-095c-4233-8206-147d729428cd-kube-api-access-fnkp9\") pod \"maas-api-revoke-sub-4e181c48-wr8jt\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:18.061925 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:18.061893    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:18.346501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:18.346470    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"]
Nov 02 01:48:18.347555 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:48:18.347526    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999c26e2_095c_4233_8206_147d729428cd.slice/crio-9f357b5198da6ca6e18e0abf38b227e0df5e25931c5b034f53e632952622e17e WatchSource:0}: Error finding container 9f357b5198da6ca6e18e0abf38b227e0df5e25931c5b034f53e632952622e17e: Status 404 returned error can't find the container with id 9f357b5198da6ca6e18e0abf38b227e0df5e25931c5b034f53e632952622e17e
Nov 02 01:48:19.057683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:19.057648    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt" event={"ID":"999c26e2-095c-4233-8206-147d729428cd","Type":"ContainerStarted","Data":"d65edc61c6146b2da8875fbd8609d99fd5e51e8b479ee936f37c1cba93071730"}
Nov 02 01:48:19.058687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:19.058662    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt" event={"ID":"999c26e2-095c-4233-8206-147d729428cd","Type":"ContainerStarted","Data":"9f357b5198da6ca6e18e0abf38b227e0df5e25931c5b034f53e632952622e17e"}
Nov 02 01:48:19.088314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:19.087440    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt" podStartSLOduration=2.087420293 podStartE2EDuration="2.087420293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:19.085354839 +0000 UTC m=+2004.390287667" watchObservedRunningTime="2026-11-02 01:48:19.087420293 +0000 UTC m=+2004.392353123"
Nov 02 01:48:20.067450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:20.067410    4851 generic.go:356] "Generic (PLEG): container finished" podID="999c26e2-095c-4233-8206-147d729428cd" containerID="d65edc61c6146b2da8875fbd8609d99fd5e51e8b479ee936f37c1cba93071730" exitCode=0
Nov 02 01:48:20.067959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:20.067499    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt" event={"ID":"999c26e2-095c-4233-8206-147d729428cd","Type":"ContainerDied","Data":"d65edc61c6146b2da8875fbd8609d99fd5e51e8b479ee936f37c1cba93071730"}
Nov 02 01:48:21.279681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:21.279652    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:21.445374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:21.439219    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999c26e2-095c-4233-8206-147d729428cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c26e2-095c-4233-8206-147d729428cd-maas-api-openshift-ca\") pod \"999c26e2-095c-4233-8206-147d729428cd\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") "
Nov 02 01:48:21.445374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:21.439341    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c26e2-095c-4233-8206-147d729428cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c26e2-095c-4233-8206-147d729428cd-maas-api-xks-ca\") pod \"999c26e2-095c-4233-8206-147d729428cd\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") "
Nov 02 01:48:21.445374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:21.439388    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999c26e2-095c-4233-8206-147d729428cd-kube-api-access-fnkp9\" (UniqueName: \"kubernetes.io/projected/999c26e2-095c-4233-8206-147d729428cd-kube-api-access-fnkp9\") pod \"999c26e2-095c-4233-8206-147d729428cd\" (UID: \"999c26e2-095c-4233-8206-147d729428cd\") "
Nov 02 01:48:21.445374 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:21.440863    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999c26e2-095c-4233-8206-147d729428cd-maas-api-openshift-ca" pod "999c26e2-095c-4233-8206-147d729428cd" (UID: "999c26e2-095c-4233-8206-147d729428cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:21.464536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:21.462074    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999c26e2-095c-4233-8206-147d729428cd-kube-api-access-fnkp9" pod "999c26e2-095c-4233-8206-147d729428cd" (UID: "999c26e2-095c-4233-8206-147d729428cd"). InnerVolumeSpecName "kube-api-access-fnkp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:21.467327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:21.465957    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c26e2-095c-4233-8206-147d729428cd-maas-api-xks-ca" pod "999c26e2-095c-4233-8206-147d729428cd" (UID: "999c26e2-095c-4233-8206-147d729428cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:21.541159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:21.541121    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c26e2-095c-4233-8206-147d729428cd-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:21.541159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:21.541154    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c26e2-095c-4233-8206-147d729428cd-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:21.541159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:21.541163    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnkp9\" (UniqueName: \"kubernetes.io/projected/999c26e2-095c-4233-8206-147d729428cd-kube-api-access-fnkp9\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:22.083883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:22.083854    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"
Nov 02 01:48:22.084073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:22.083893    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt" event={"ID":"999c26e2-095c-4233-8206-147d729428cd","Type":"ContainerDied","Data":"9f357b5198da6ca6e18e0abf38b227e0df5e25931c5b034f53e632952622e17e"}
Nov 02 01:48:22.084073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:22.083945    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f357b5198da6ca6e18e0abf38b227e0df5e25931c5b034f53e632952622e17e"
Nov 02 01:48:30.836274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:30.836240    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"]
Nov 02 01:48:31.078279 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.078224    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"]
Nov 02 01:48:31.078556 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.078516    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy" containerID="cri-o://0d0888dfa1c0c908d9717a0f95aebdb8a7a14bccebd487e7e15648342c542141" gracePeriod=30
Nov 02 01:48:31.640589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.640551    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"]
Nov 02 01:48:31.641123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.641109    4851 cpu_manager.go:414] "containerMap: removing container" podUID="999c26e2-095c-4233-8206-147d729428cd" containerName="revoke-keys"
Nov 02 01:48:31.641171 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.641124    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c26e2-095c-4233-8206-147d729428cd" containerName="revoke-keys"
Nov 02 01:48:31.641216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.641209    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c26e2-095c-4233-8206-147d729428cd" containerName="revoke-keys"
Nov 02 01:48:31.645140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.645123    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:31.649179 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.649151    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:31.649473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.649221    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:48:31.655333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.654405    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"]
Nov 02 01:48:31.725267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.725220    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:31.725519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.725402    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:31.725590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.725540    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vzf\" (UniqueName: \"kubernetes.io/projected/1f3e2758-12a5-49ca-9b26-755976c576c5-kube-api-access-96vzf\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:31.827504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.827457    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vzf\" (UniqueName: \"kubernetes.io/projected/1f3e2758-12a5-49ca-9b26-755976c576c5-kube-api-access-96vzf\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:31.827728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.827631    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:31.827728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.827722    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:31.828376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.828355    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:31.830562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.830528    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:31.840151 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.840123    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vzf\" (UniqueName: \"kubernetes.io/projected/1f3e2758-12a5-49ca-9b26-755976c576c5-kube-api-access-96vzf\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:31.962229 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:31.962199    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:32.257070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:32.257039    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"]
Nov 02 01:48:33.267731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:33.267700    4851 generic.go:356] "Generic (PLEG): container finished" podID="1f3e2758-12a5-49ca-9b26-755976c576c5" containerID="68574e939f898f47340e1f0717d6fc4b92afd785d339e423368b5b7036b3858c" exitCode=0
Nov 02 01:48:33.268212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:33.267773    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx" event={"ID":"1f3e2758-12a5-49ca-9b26-755976c576c5","Type":"ContainerDied","Data":"68574e939f898f47340e1f0717d6fc4b92afd785d339e423368b5b7036b3858c"}
Nov 02 01:48:33.268212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:33.267826    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx" event={"ID":"1f3e2758-12a5-49ca-9b26-755976c576c5","Type":"ContainerStarted","Data":"d99a598511373b8e4d45842df4a65e8bed1666e31ba3049d2eaab4ca87f80952"}
Nov 02 01:48:34.450259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:34.450236    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:34.497782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:34.497753    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3e2758-12a5-49ca-9b26-755976c576c5-kube-api-access-96vzf\" (UniqueName: \"kubernetes.io/projected/1f3e2758-12a5-49ca-9b26-755976c576c5-kube-api-access-96vzf\") pod \"1f3e2758-12a5-49ca-9b26-755976c576c5\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") "
Nov 02 01:48:34.497933 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:34.497788    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-openshift-ca\") pod \"1f3e2758-12a5-49ca-9b26-755976c576c5\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") "
Nov 02 01:48:34.498271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:34.498238    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-openshift-ca" pod "1f3e2758-12a5-49ca-9b26-755976c576c5" (UID: "1f3e2758-12a5-49ca-9b26-755976c576c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:34.500308 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:34.500275    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3e2758-12a5-49ca-9b26-755976c576c5-kube-api-access-96vzf" pod "1f3e2758-12a5-49ca-9b26-755976c576c5" (UID: "1f3e2758-12a5-49ca-9b26-755976c576c5"). InnerVolumeSpecName "kube-api-access-96vzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:34.598782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:34.598705    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-xks-ca\") pod \"1f3e2758-12a5-49ca-9b26-755976c576c5\" (UID: \"1f3e2758-12a5-49ca-9b26-755976c576c5\") "
Nov 02 01:48:34.599148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:34.599121    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96vzf\" (UniqueName: \"kubernetes.io/projected/1f3e2758-12a5-49ca-9b26-755976c576c5-kube-api-access-96vzf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:34.599148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:34.599142    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:34.600960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:34.600936    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-xks-ca" pod "1f3e2758-12a5-49ca-9b26-755976c576c5" (UID: "1f3e2758-12a5-49ca-9b26-755976c576c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:34.699976 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:34.699949    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3e2758-12a5-49ca-9b26-755976c576c5-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:35.286756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:35.286721    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"
Nov 02 01:48:35.286756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:35.286748    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx" event={"ID":"1f3e2758-12a5-49ca-9b26-755976c576c5","Type":"ContainerDied","Data":"d99a598511373b8e4d45842df4a65e8bed1666e31ba3049d2eaab4ca87f80952"}
Nov 02 01:48:35.286977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:35.286806    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d99a598511373b8e4d45842df4a65e8bed1666e31ba3049d2eaab4ca87f80952"
Nov 02 01:48:36.300093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.300058    4851 generic.go:356] "Generic (PLEG): container finished" podID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerID="0d0888dfa1c0c908d9717a0f95aebdb8a7a14bccebd487e7e15648342c542141" exitCode=0
Nov 02 01:48:36.300637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.300091    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" event={"ID":"5e310f6b-1d42-4471-a2a2-3562f8199803","Type":"ContainerDied","Data":"0d0888dfa1c0c908d9717a0f95aebdb8a7a14bccebd487e7e15648342c542141"}
Nov 02 01:48:36.300637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.300164    4851 scope.go:122] "RemoveContainer" containerID="cc3967322c448e22c1ec6f9ce1284315850dfe653741ac4d0aa355da5cbc713b"
Nov 02 01:48:36.383301 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.383261    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:48:36.430853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.430812    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-data\") pod \"5e310f6b-1d42-4471-a2a2-3562f8199803\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") "
Nov 02 01:48:36.431049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.430890    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e310f6b-1d42-4471-a2a2-3562f8199803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e310f6b-1d42-4471-a2a2-3562f8199803-istiod-ca-cert\") pod \"5e310f6b-1d42-4471-a2a2-3562f8199803\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") "
Nov 02 01:48:36.431049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.430928    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-envoy\") pod \"5e310f6b-1d42-4471-a2a2-3562f8199803\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") "
Nov 02 01:48:36.431049 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.430969    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-token\" (UniqueName: \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-token\") pod \"5e310f6b-1d42-4471-a2a2-3562f8199803\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") "
Nov 02 01:48:36.431250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431091    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-kube-api-access-r7rhf\" (UniqueName: \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-kube-api-access-r7rhf\") pod \"5e310f6b-1d42-4471-a2a2-3562f8199803\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") "
Nov 02 01:48:36.431250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431138    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-credential-socket\") pod \"5e310f6b-1d42-4471-a2a2-3562f8199803\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") "
Nov 02 01:48:36.431250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431177    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-socket\") pod \"5e310f6b-1d42-4471-a2a2-3562f8199803\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") "
Nov 02 01:48:36.431250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431192    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-data" pod "5e310f6b-1d42-4471-a2a2-3562f8199803" (UID: "5e310f6b-1d42-4471-a2a2-3562f8199803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:36.431250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431214    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-podinfo\") pod \"5e310f6b-1d42-4471-a2a2-3562f8199803\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") "
Nov 02 01:48:36.431586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431264    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e310f6b-1d42-4471-a2a2-3562f8199803-istiod-ca-cert" pod "5e310f6b-1d42-4471-a2a2-3562f8199803" (UID: "5e310f6b-1d42-4471-a2a2-3562f8199803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:36.431586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431462    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-credential-socket" pod "5e310f6b-1d42-4471-a2a2-3562f8199803" (UID: "5e310f6b-1d42-4471-a2a2-3562f8199803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:36.431586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431567    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-socket" pod "5e310f6b-1d42-4471-a2a2-3562f8199803" (UID: "5e310f6b-1d42-4471-a2a2-3562f8199803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:36.431816 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431795    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:36.431890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431822    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:36.431890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431839    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:36.431890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.431853    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e310f6b-1d42-4471-a2a2-3562f8199803-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:36.434521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.434459    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-envoy" pod "5e310f6b-1d42-4471-a2a2-3562f8199803" (UID: "5e310f6b-1d42-4471-a2a2-3562f8199803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:36.434521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.434482    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-podinfo" pod "5e310f6b-1d42-4471-a2a2-3562f8199803" (UID: "5e310f6b-1d42-4471-a2a2-3562f8199803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:48:36.434521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.434497    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-kube-api-access-r7rhf" pod "5e310f6b-1d42-4471-a2a2-3562f8199803" (UID: "5e310f6b-1d42-4471-a2a2-3562f8199803"). InnerVolumeSpecName "kube-api-access-r7rhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:36.435086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.435066    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-token" pod "5e310f6b-1d42-4471-a2a2-3562f8199803" (UID: "5e310f6b-1d42-4471-a2a2-3562f8199803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:36.534661 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.534619    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-certs\") pod \"5e310f6b-1d42-4471-a2a2-3562f8199803\" (UID: \"5e310f6b-1d42-4471-a2a2-3562f8199803\") "
Nov 02 01:48:36.535586 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.535562    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:36.535656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.535599    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:36.535656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.535615    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rhf\" (UniqueName: \"kubernetes.io/projected/5e310f6b-1d42-4471-a2a2-3562f8199803-kube-api-access-r7rhf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:36.535656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.535628    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e310f6b-1d42-4471-a2a2-3562f8199803-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:36.535920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.535899    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-certs" pod "5e310f6b-1d42-4471-a2a2-3562f8199803" (UID: "5e310f6b-1d42-4471-a2a2-3562f8199803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:36.636382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:36.636272    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e310f6b-1d42-4471-a2a2-3562f8199803-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:37.308705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:37.308678    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"
Nov 02 01:48:37.309139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:37.308707    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn" event={"ID":"5e310f6b-1d42-4471-a2a2-3562f8199803","Type":"ContainerDied","Data":"7356af1585d8c69397c0423c8bc775a3f1e1d9974c89c88d56524525ed9ebcd2"}
Nov 02 01:48:37.309139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:37.308770    4851 scope.go:122] "RemoveContainer" containerID="0d0888dfa1c0c908d9717a0f95aebdb8a7a14bccebd487e7e15648342c542141"
Nov 02 01:48:37.330423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:37.330389    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"]
Nov 02 01:48:37.333843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:37.333818    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-hmrvn"]
Nov 02 01:48:38.846498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:38.846468    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" path="/var/lib/kubelet/pods/5e310f6b-1d42-4471-a2a2-3562f8199803/volumes"
Nov 02 01:48:41.196083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:41.196045    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"]
Nov 02 01:48:41.209760 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:41.209731    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"]
Nov 02 01:48:41.581877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:41.581762    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"]
Nov 02 01:48:41.582397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:41.582362    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv" podUID="96064425-7a87-45ac-83a7-36fba9bb924d" containerName="maas-api" containerID="cri-o://fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6" gracePeriod=30
Nov 02 01:48:41.907654 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:41.907621    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:48:42.025521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.025482    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-maas-api-tls\") pod \"96064425-7a87-45ac-83a7-36fba9bb924d\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") "
Nov 02 01:48:42.025713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.025565    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-metrics-certs\") pod \"96064425-7a87-45ac-83a7-36fba9bb924d\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") "
Nov 02 01:48:42.025713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.025638    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96064425-7a87-45ac-83a7-36fba9bb924d-kube-api-access-62wz8\" (UniqueName: \"kubernetes.io/projected/96064425-7a87-45ac-83a7-36fba9bb924d-kube-api-access-62wz8\") pod \"96064425-7a87-45ac-83a7-36fba9bb924d\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") "
Nov 02 01:48:42.025713 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.025656    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96064425-7a87-45ac-83a7-36fba9bb924d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96064425-7a87-45ac-83a7-36fba9bb924d-openshift-service-ca\") pod \"96064425-7a87-45ac-83a7-36fba9bb924d\" (UID: \"96064425-7a87-45ac-83a7-36fba9bb924d\") "
Nov 02 01:48:42.026126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.026084    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96064425-7a87-45ac-83a7-36fba9bb924d-openshift-service-ca" pod "96064425-7a87-45ac-83a7-36fba9bb924d" (UID: "96064425-7a87-45ac-83a7-36fba9bb924d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:42.027875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.027841    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-maas-api-tls" pod "96064425-7a87-45ac-83a7-36fba9bb924d" (UID: "96064425-7a87-45ac-83a7-36fba9bb924d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:42.027999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.027873    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-metrics-certs" pod "96064425-7a87-45ac-83a7-36fba9bb924d" (UID: "96064425-7a87-45ac-83a7-36fba9bb924d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:42.028059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.027991    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96064425-7a87-45ac-83a7-36fba9bb924d-kube-api-access-62wz8" pod "96064425-7a87-45ac-83a7-36fba9bb924d" (UID: "96064425-7a87-45ac-83a7-36fba9bb924d"). InnerVolumeSpecName "kube-api-access-62wz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:42.140263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.139281    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-maas-api-tls\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:42.140263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.139341    4851 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96064425-7a87-45ac-83a7-36fba9bb924d-metrics-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:42.140263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.139358    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62wz8\" (UniqueName: \"kubernetes.io/projected/96064425-7a87-45ac-83a7-36fba9bb924d-kube-api-access-62wz8\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:42.140263 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.139371    4851 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96064425-7a87-45ac-83a7-36fba9bb924d-openshift-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:42.392463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.392366    4851 generic.go:356] "Generic (PLEG): container finished" podID="96064425-7a87-45ac-83a7-36fba9bb924d" containerID="fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6" exitCode=0
Nov 02 01:48:42.392463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.392434    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"
Nov 02 01:48:42.393019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.392459    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv" event={"ID":"96064425-7a87-45ac-83a7-36fba9bb924d","Type":"ContainerDied","Data":"fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6"}
Nov 02 01:48:42.393019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.392555    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv" event={"ID":"96064425-7a87-45ac-83a7-36fba9bb924d","Type":"ContainerDied","Data":"8c64064bc5b2887c88a5e65e4216e2921f8dc97dc4a4dd0d8093a2f7dfa6cb25"}
Nov 02 01:48:42.393019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.392587    4851 scope.go:122] "RemoveContainer" containerID="fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6"
Nov 02 01:48:42.405165 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.405138    4851 scope.go:122] "RemoveContainer" containerID="fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6"
Nov 02 01:48:42.405545 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:48:42.405519    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6\": container with ID starting with fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6 not found: ID does not exist" containerID="fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6"
Nov 02 01:48:42.405653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.405553    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6"} err="failed to get container status \"fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6\": rpc error: code = NotFound desc = could not find container \"fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6\": container with ID starting with fb3781d0b04e1a9b9355205d38d4329cfc00e2aaaf978c46ba0fa317b53d37f6 not found: ID does not exist"
Nov 02 01:48:42.453612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.453569    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"]
Nov 02 01:48:42.476678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.476640    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3c4ea5-8577466b8c-5q2nv"]
Nov 02 01:48:42.867054 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:42.867005    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96064425-7a87-45ac-83a7-36fba9bb924d" path="/var/lib/kubelet/pods/96064425-7a87-45ac-83a7-36fba9bb924d/volumes"
Nov 02 01:48:45.858794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:45.858754    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l" podUID="e713d800-8e9d-4d87-8b1d-8d039c1d379b" containerName="main" containerID="cri-o://cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2" gracePeriod=45
Nov 02 01:48:46.129525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.129457    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:48:46.196837 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.196789    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm" podUID="11a2b0ca-9b50-4b68-b0bb-7d7425bd3655" containerName="payload-processing" containerID="cri-o://a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c" gracePeriod=25
Nov 02 01:48:46.203804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.203771    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e713d800-8e9d-4d87-8b1d-8d039c1d379b-kube-api-access-5x4jh\" (UniqueName: \"kubernetes.io/projected/e713d800-8e9d-4d87-8b1d-8d039c1d379b-kube-api-access-5x4jh\") pod \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") "
Nov 02 01:48:46.203952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.203841    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-dshm\") pod \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") "
Nov 02 01:48:46.203994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.203948    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tmp-dir\") pod \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") "
Nov 02 01:48:46.204062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.204044    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-home\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-home\") pod \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") "
Nov 02 01:48:46.204108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.204096    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tls-certs\") pod \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") "
Nov 02 01:48:46.204251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.204230    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-model-cache\") pod \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\" (UID: \"e713d800-8e9d-4d87-8b1d-8d039c1d379b\") "
Nov 02 01:48:46.204506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.204432    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tmp-dir" pod "e713d800-8e9d-4d87-8b1d-8d039c1d379b" (UID: "e713d800-8e9d-4d87-8b1d-8d039c1d379b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:46.204693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.204674    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-model-cache" pod "e713d800-8e9d-4d87-8b1d-8d039c1d379b" (UID: "e713d800-8e9d-4d87-8b1d-8d039c1d379b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:46.204918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.204897    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-home" pod "e713d800-8e9d-4d87-8b1d-8d039c1d379b" (UID: "e713d800-8e9d-4d87-8b1d-8d039c1d379b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:46.205097 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.205081    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:46.205148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.205103    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:46.205148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.205116    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:46.206899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.206873    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-dshm" pod "e713d800-8e9d-4d87-8b1d-8d039c1d379b" (UID: "e713d800-8e9d-4d87-8b1d-8d039c1d379b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:46.207038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.206893    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e713d800-8e9d-4d87-8b1d-8d039c1d379b-kube-api-access-5x4jh" pod "e713d800-8e9d-4d87-8b1d-8d039c1d379b" (UID: "e713d800-8e9d-4d87-8b1d-8d039c1d379b"). InnerVolumeSpecName "kube-api-access-5x4jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:46.207113 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.207047    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tls-certs" pod "e713d800-8e9d-4d87-8b1d-8d039c1d379b" (UID: "e713d800-8e9d-4d87-8b1d-8d039c1d379b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:46.210159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.210125    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq" podUID="49ae4474-4f10-47b8-86a3-d87b2f6e6021" containerName="payload-pre-processing" containerID="cri-o://c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef" gracePeriod=25
Nov 02 01:48:46.306136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.306101    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e713d800-8e9d-4d87-8b1d-8d039c1d379b-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:46.306136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.306132    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x4jh\" (UniqueName: \"kubernetes.io/projected/e713d800-8e9d-4d87-8b1d-8d039c1d379b-kube-api-access-5x4jh\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:46.306136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.306142    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e713d800-8e9d-4d87-8b1d-8d039c1d379b-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:46.496367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.484252    4851 generic.go:356] "Generic (PLEG): container finished" podID="e713d800-8e9d-4d87-8b1d-8d039c1d379b" containerID="cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2" exitCode=0
Nov 02 01:48:46.496367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.484343    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l" event={"ID":"e713d800-8e9d-4d87-8b1d-8d039c1d379b","Type":"ContainerDied","Data":"cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2"}
Nov 02 01:48:46.496367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.484383    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l" event={"ID":"e713d800-8e9d-4d87-8b1d-8d039c1d379b","Type":"ContainerDied","Data":"a364ac649c538e53a06e38840c698bc5fa688ff970bbb93ba57d1221c18fca1d"}
Nov 02 01:48:46.496367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.484407    4851 scope.go:122] "RemoveContainer" containerID="cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2"
Nov 02 01:48:46.496367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.484538    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"
Nov 02 01:48:46.569756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.569693    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"]
Nov 02 01:48:46.571156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.571133    4851 scope.go:122] "RemoveContainer" containerID="cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2"
Nov 02 01:48:46.571679 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:48:46.571604    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2\": container with ID starting with cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2 not found: ID does not exist" containerID="cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2"
Nov 02 01:48:46.571679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.571649    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2"} err="failed to get container status \"cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2\": rpc error: code = NotFound desc = could not find container \"cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2\": container with ID starting with cea8247166e4e909b37bf75c3732382396e627f7eaa1af53a93c914dabcd16c2 not found: ID does not exist"
Nov 02 01:48:46.573600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.573576    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d4a0/auth-test-model-c8d4a0-kserve-dcddcb4f-7c67l"]
Nov 02 01:48:46.591178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.591149    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"]
Nov 02 01:48:46.594002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.593976    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0e06f2-6r2tx"]
Nov 02 01:48:46.848877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.848789    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3e2758-12a5-49ca-9b26-755976c576c5" path="/var/lib/kubelet/pods/1f3e2758-12a5-49ca-9b26-755976c576c5/volumes"
Nov 02 01:48:46.849394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:46.849366    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e713d800-8e9d-4d87-8b1d-8d039c1d379b" path="/var/lib/kubelet/pods/e713d800-8e9d-4d87-8b1d-8d039c1d379b/volumes"
Nov 02 01:48:47.656427 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:48:47.656387    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:48:48.480364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:48.480332    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"]
Nov 02 01:48:48.492746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:48.492714    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"]
Nov 02 01:48:49.425089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.425058    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"]
Nov 02 01:48:49.425590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.425388    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy" containerID="cri-o://6a16f6aa768b57d1737e5f51a12cf40e31d00f16f8faaa7e2ba86c1610b9216c" gracePeriod=30
Nov 02 01:48:49.929693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.929661    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"]
Nov 02 01:48:49.930205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930191    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy"
Nov 02 01:48:49.930257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930207    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy"
Nov 02 01:48:49.930257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930237    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy"
Nov 02 01:48:49.930257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930242    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy"
Nov 02 01:48:49.930257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930257    4851 cpu_manager.go:414] "containerMap: removing container" podUID="96064425-7a87-45ac-83a7-36fba9bb924d" containerName="maas-api"
Nov 02 01:48:49.930412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930262    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96064425-7a87-45ac-83a7-36fba9bb924d" containerName="maas-api"
Nov 02 01:48:49.930412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930269    4851 cpu_manager.go:414] "containerMap: removing container" podUID="e713d800-8e9d-4d87-8b1d-8d039c1d379b" containerName="main"
Nov 02 01:48:49.930412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930274    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e713d800-8e9d-4d87-8b1d-8d039c1d379b" containerName="main"
Nov 02 01:48:49.930412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930283    4851 cpu_manager.go:414] "containerMap: removing container" podUID="1f3e2758-12a5-49ca-9b26-755976c576c5" containerName="revoke-keys"
Nov 02 01:48:49.930412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930298    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3e2758-12a5-49ca-9b26-755976c576c5" containerName="revoke-keys"
Nov 02 01:48:49.930412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930391    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy"
Nov 02 01:48:49.930412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930408    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e310f6b-1d42-4471-a2a2-3562f8199803" containerName="istio-proxy"
Nov 02 01:48:49.930630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930420    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="96064425-7a87-45ac-83a7-36fba9bb924d" containerName="maas-api"
Nov 02 01:48:49.930630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930429    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="e713d800-8e9d-4d87-8b1d-8d039c1d379b" containerName="main"
Nov 02 01:48:49.930630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.930435    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3e2758-12a5-49ca-9b26-755976c576c5" containerName="revoke-keys"
Nov 02 01:48:49.933948 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.933929    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:49.943225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.943196    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"]
Nov 02 01:48:49.966648 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.966624    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:49.967131 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:49.967108    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:48:50.079376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.079341    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpfbw\" (UniqueName: \"kubernetes.io/projected/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-kube-api-access-gpfbw\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:50.079376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.079381    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:50.079611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.079561    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:50.180700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.180628    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:50.180876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.180739    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpfbw\" (UniqueName: \"kubernetes.io/projected/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-kube-api-access-gpfbw\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:50.180876 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.180767    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:50.181523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.181457    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:50.183187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.183161    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:50.193050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.193027    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpfbw\" (UniqueName: \"kubernetes.io/projected/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-kube-api-access-gpfbw\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:50.261136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.261107    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:50.549492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.549466    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"]
Nov 02 01:48:50.551247 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:48:50.551215    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e6c7c4_5f87_4295_91f2_7f0a4795bc06.slice/crio-d9ff7de2f3fef330b5fc720855942cf81c5f5a95b623d4983c9498eb2bfbbcb6 WatchSource:0}: Error finding container d9ff7de2f3fef330b5fc720855942cf81c5f5a95b623d4983c9498eb2bfbbcb6: Status 404 returned error can't find the container with id d9ff7de2f3fef330b5fc720855942cf81c5f5a95b623d4983c9498eb2bfbbcb6
Nov 02 01:48:50.583540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:50.571441    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2" event={"ID":"70e6c7c4-5f87-4295-91f2-7f0a4795bc06","Type":"ContainerStarted","Data":"d9ff7de2f3fef330b5fc720855942cf81c5f5a95b623d4983c9498eb2bfbbcb6"}
Nov 02 01:48:51.581364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:51.581281    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2" event={"ID":"70e6c7c4-5f87-4295-91f2-7f0a4795bc06","Type":"ContainerStarted","Data":"e337b0e2a0ebc68456d47ee3830a15d4c551daf80590e5fd1d1e6a383ae46290"}
Nov 02 01:48:51.598347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:51.598313    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"]
Nov 02 01:48:51.598730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:51.598586    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564" podUID="19c91b78-6c4c-4802-95e8-c9a80956681b" containerName="maas-api" containerID="cri-o://1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57" gracePeriod=30
Nov 02 01:48:51.604585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:51.604531    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2" podStartSLOduration=2.604513991 podStartE2EDuration="2.604513991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:51.603140565 +0000 UTC m=+2036.908073392" watchObservedRunningTime="2026-11-02 01:48:51.604513991 +0000 UTC m=+2036.909446820"
Nov 02 01:48:52.106814 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.106791    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:48:52.147092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.147021    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c91b78-6c4c-4802-95e8-c9a80956681b-kube-api-access-xpzm2\" (UniqueName: \"kubernetes.io/projected/19c91b78-6c4c-4802-95e8-c9a80956681b-kube-api-access-xpzm2\") pod \"19c91b78-6c4c-4802-95e8-c9a80956681b\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") "
Nov 02 01:48:52.147258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.147147    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c91b78-6c4c-4802-95e8-c9a80956681b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c91b78-6c4c-4802-95e8-c9a80956681b-openshift-service-ca\") pod \"19c91b78-6c4c-4802-95e8-c9a80956681b\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") "
Nov 02 01:48:52.147258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.147212    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-maas-api-tls\") pod \"19c91b78-6c4c-4802-95e8-c9a80956681b\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") "
Nov 02 01:48:52.147258 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.147242    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-metrics-certs\") pod \"19c91b78-6c4c-4802-95e8-c9a80956681b\" (UID: \"19c91b78-6c4c-4802-95e8-c9a80956681b\") "
Nov 02 01:48:52.147608 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.147576    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c91b78-6c4c-4802-95e8-c9a80956681b-openshift-service-ca" pod "19c91b78-6c4c-4802-95e8-c9a80956681b" (UID: "19c91b78-6c4c-4802-95e8-c9a80956681b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:52.147762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.147744    4851 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c91b78-6c4c-4802-95e8-c9a80956681b-openshift-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:52.149531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.149502    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-metrics-certs" pod "19c91b78-6c4c-4802-95e8-c9a80956681b" (UID: "19c91b78-6c4c-4802-95e8-c9a80956681b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:52.149644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.149511    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c91b78-6c4c-4802-95e8-c9a80956681b-kube-api-access-xpzm2" pod "19c91b78-6c4c-4802-95e8-c9a80956681b" (UID: "19c91b78-6c4c-4802-95e8-c9a80956681b"). InnerVolumeSpecName "kube-api-access-xpzm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:52.149843 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.149822    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-maas-api-tls" pod "19c91b78-6c4c-4802-95e8-c9a80956681b" (UID: "19c91b78-6c4c-4802-95e8-c9a80956681b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:52.248521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.248488    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpzm2\" (UniqueName: \"kubernetes.io/projected/19c91b78-6c4c-4802-95e8-c9a80956681b-kube-api-access-xpzm2\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:52.248521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.248521    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-maas-api-tls\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:52.248745 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.248532    4851 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19c91b78-6c4c-4802-95e8-c9a80956681b-metrics-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:52.589373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.589338    4851 generic.go:356] "Generic (PLEG): container finished" podID="19c91b78-6c4c-4802-95e8-c9a80956681b" containerID="1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57" exitCode=0
Nov 02 01:48:52.589886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.589382    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"
Nov 02 01:48:52.589886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.589407    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564" event={"ID":"19c91b78-6c4c-4802-95e8-c9a80956681b","Type":"ContainerDied","Data":"1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57"}
Nov 02 01:48:52.589886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.589468    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564" event={"ID":"19c91b78-6c4c-4802-95e8-c9a80956681b","Type":"ContainerDied","Data":"750d655318a4ad94de7d6d6609abafb511923873b3250211f0627cc8ae64cc48"}
Nov 02 01:48:52.589886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.589491    4851 scope.go:122] "RemoveContainer" containerID="1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57"
Nov 02 01:48:52.591559 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.591536    4851 generic.go:356] "Generic (PLEG): container finished" podID="70e6c7c4-5f87-4295-91f2-7f0a4795bc06" containerID="e337b0e2a0ebc68456d47ee3830a15d4c551daf80590e5fd1d1e6a383ae46290" exitCode=0
Nov 02 01:48:52.591688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.591595    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2" event={"ID":"70e6c7c4-5f87-4295-91f2-7f0a4795bc06","Type":"ContainerDied","Data":"e337b0e2a0ebc68456d47ee3830a15d4c551daf80590e5fd1d1e6a383ae46290"}
Nov 02 01:48:52.601780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.601592    4851 scope.go:122] "RemoveContainer" containerID="1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57"
Nov 02 01:48:52.601910 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:48:52.601891    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57\": container with ID starting with 1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57 not found: ID does not exist" containerID="1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57"
Nov 02 01:48:52.601957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.601919    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57"} err="failed to get container status \"1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57\": rpc error: code = NotFound desc = could not find container \"1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57\": container with ID starting with 1a73352cb6ee0423b4b6f2606a381189cfc283569dcdd670b75f8c8a763cdb57 not found: ID does not exist"
Nov 02 01:48:52.626852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.626825    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"]
Nov 02 01:48:52.630487 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.630464    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-eaec8c-6869c4cd66-f4564"]
Nov 02 01:48:52.847553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:52.847482    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c91b78-6c4c-4802-95e8-c9a80956681b" path="/var/lib/kubelet/pods/19c91b78-6c4c-4802-95e8-c9a80956681b/volumes"
Nov 02 01:48:53.322223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.322188    4851 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:48:53.322418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.322244    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:48:53.481305 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.481251    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g" podUID="31addff9-dc30-4d8a-8753-8229f834003f" containerName="payload-processing" containerID="cri-o://e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65" gracePeriod=25
Nov 02 01:48:53.493965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.493931    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57" podUID="77229192-c71d-4a53-becc-3338c7de9c81" containerName="payload-pre-processing" containerID="cri-o://a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b" gracePeriod=25
Nov 02 01:48:53.766260 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.766230    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:53.770473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.770449    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-kube-api-access-gpfbw\" (UniqueName: \"kubernetes.io/projected/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-kube-api-access-gpfbw\") pod \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") "
Nov 02 01:48:53.770681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.770661    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-openshift-ca\") pod \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") "
Nov 02 01:48:53.770741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.770705    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-xks-ca\") pod \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\" (UID: \"70e6c7c4-5f87-4295-91f2-7f0a4795bc06\") "
Nov 02 01:48:53.770983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.770960    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-openshift-ca" pod "70e6c7c4-5f87-4295-91f2-7f0a4795bc06" (UID: "70e6c7c4-5f87-4295-91f2-7f0a4795bc06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:53.771116 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.771076    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:53.772831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.772809    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-kube-api-access-gpfbw" pod "70e6c7c4-5f87-4295-91f2-7f0a4795bc06" (UID: "70e6c7c4-5f87-4295-91f2-7f0a4795bc06"). InnerVolumeSpecName "kube-api-access-gpfbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:53.772945 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.772931    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-xks-ca" pod "70e6c7c4-5f87-4295-91f2-7f0a4795bc06" (UID: "70e6c7c4-5f87-4295-91f2-7f0a4795bc06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:53.871993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.871906    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:53.871993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:53.871933    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpfbw\" (UniqueName: \"kubernetes.io/projected/70e6c7c4-5f87-4295-91f2-7f0a4795bc06-kube-api-access-gpfbw\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.621872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.621843    4851 generic.go:356] "Generic (PLEG): container finished" podID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerID="6a16f6aa768b57d1737e5f51a12cf40e31d00f16f8faaa7e2ba86c1610b9216c" exitCode=0
Nov 02 01:48:54.622040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.621919    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" event={"ID":"5377e2b2-cc37-4884-96a0-9a5e2628d968","Type":"ContainerDied","Data":"6a16f6aa768b57d1737e5f51a12cf40e31d00f16f8faaa7e2ba86c1610b9216c"}
Nov 02 01:48:54.622040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.622010    4851 scope.go:122] "RemoveContainer" containerID="425c1575a0d8a761f7b21ee2d9b69e7b07dddbcc0c7656a5454786f8eab9141f"
Nov 02 01:48:54.623487 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.623466    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"
Nov 02 01:48:54.623611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.623493    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2" event={"ID":"70e6c7c4-5f87-4295-91f2-7f0a4795bc06","Type":"ContainerDied","Data":"d9ff7de2f3fef330b5fc720855942cf81c5f5a95b623d4983c9498eb2bfbbcb6"}
Nov 02 01:48:54.623611 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.623562    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9ff7de2f3fef330b5fc720855942cf81c5f5a95b623d4983c9498eb2bfbbcb6"
Nov 02 01:48:54.691415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.691360    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:48:54.766184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.766145    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"]
Nov 02 01:48:54.767364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767334    4851 cpu_manager.go:414] "containerMap: removing container" podUID="70e6c7c4-5f87-4295-91f2-7f0a4795bc06" containerName="revoke-keys"
Nov 02 01:48:54.767364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767361    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e6c7c4-5f87-4295-91f2-7f0a4795bc06" containerName="revoke-keys"
Nov 02 01:48:54.767898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767389    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy"
Nov 02 01:48:54.767898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767397    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy"
Nov 02 01:48:54.767898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767410    4851 cpu_manager.go:414] "containerMap: removing container" podUID="19c91b78-6c4c-4802-95e8-c9a80956681b" containerName="maas-api"
Nov 02 01:48:54.767898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767418    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c91b78-6c4c-4802-95e8-c9a80956681b" containerName="maas-api"
Nov 02 01:48:54.767898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767439    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy"
Nov 02 01:48:54.767898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767446    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy"
Nov 02 01:48:54.767898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767781    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e6c7c4-5f87-4295-91f2-7f0a4795bc06" containerName="revoke-keys"
Nov 02 01:48:54.767898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767804    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy"
Nov 02 01:48:54.767898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.767812    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c91b78-6c4c-4802-95e8-c9a80956681b" containerName="maas-api"
Nov 02 01:48:54.774537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.774512    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:54.778649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.778618    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"]
Nov 02 01:48:54.780037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.779704    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:48:54.780037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.779722    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:48:54.780037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.779708    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:48:54.780920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.780895    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:54.803934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.803912    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-credential-socket\") pod \"5377e2b2-cc37-4884-96a0-9a5e2628d968\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") "
Nov 02 01:48:54.804051 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.803972    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-socket\") pod \"5377e2b2-cc37-4884-96a0-9a5e2628d968\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") "
Nov 02 01:48:54.804139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804106    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-data\") pod \"5377e2b2-cc37-4884-96a0-9a5e2628d968\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") "
Nov 02 01:48:54.804216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804143    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5377e2b2-cc37-4884-96a0-9a5e2628d968-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5377e2b2-cc37-4884-96a0-9a5e2628d968-istiod-ca-cert\") pod \"5377e2b2-cc37-4884-96a0-9a5e2628d968\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") "
Nov 02 01:48:54.804216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804178    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-kube-api-access-5h4q2\" (UniqueName: \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-kube-api-access-5h4q2\") pod \"5377e2b2-cc37-4884-96a0-9a5e2628d968\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") "
Nov 02 01:48:54.804394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804260    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-podinfo\") pod \"5377e2b2-cc37-4884-96a0-9a5e2628d968\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") "
Nov 02 01:48:54.804394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804277    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-socket" pod "5377e2b2-cc37-4884-96a0-9a5e2628d968" (UID: "5377e2b2-cc37-4884-96a0-9a5e2628d968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:54.804394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804310    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-certs\") pod \"5377e2b2-cc37-4884-96a0-9a5e2628d968\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") "
Nov 02 01:48:54.804394 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804377    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-token\" (UniqueName: \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-token\") pod \"5377e2b2-cc37-4884-96a0-9a5e2628d968\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") "
Nov 02 01:48:54.804622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804402    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-envoy\") pod \"5377e2b2-cc37-4884-96a0-9a5e2628d968\" (UID: \"5377e2b2-cc37-4884-96a0-9a5e2628d968\") "
Nov 02 01:48:54.804687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804376    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-data" pod "5377e2b2-cc37-4884-96a0-9a5e2628d968" (UID: "5377e2b2-cc37-4884-96a0-9a5e2628d968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:54.804687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804643    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5377e2b2-cc37-4884-96a0-9a5e2628d968-istiod-ca-cert" pod "5377e2b2-cc37-4884-96a0-9a5e2628d968" (UID: "5377e2b2-cc37-4884-96a0-9a5e2628d968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:54.804810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804795    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-certs" pod "5377e2b2-cc37-4884-96a0-9a5e2628d968" (UID: "5377e2b2-cc37-4884-96a0-9a5e2628d968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:54.804932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804914    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.805011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804938    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.805011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804953    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.805011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.804966    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5377e2b2-cc37-4884-96a0-9a5e2628d968-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.805329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.805265    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-credential-socket" pod "5377e2b2-cc37-4884-96a0-9a5e2628d968" (UID: "5377e2b2-cc37-4884-96a0-9a5e2628d968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:54.807245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.807220    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-envoy" pod "5377e2b2-cc37-4884-96a0-9a5e2628d968" (UID: "5377e2b2-cc37-4884-96a0-9a5e2628d968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:54.808264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.808245    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-podinfo" pod "5377e2b2-cc37-4884-96a0-9a5e2628d968" (UID: "5377e2b2-cc37-4884-96a0-9a5e2628d968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:48:54.808401 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.808331    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-kube-api-access-5h4q2" pod "5377e2b2-cc37-4884-96a0-9a5e2628d968" (UID: "5377e2b2-cc37-4884-96a0-9a5e2628d968"). InnerVolumeSpecName "kube-api-access-5h4q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:54.808520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.808499    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-token" pod "5377e2b2-cc37-4884-96a0-9a5e2628d968" (UID: "5377e2b2-cc37-4884-96a0-9a5e2628d968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:54.829844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.829818    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq" podUID="49ae4474-4f10-47b8-86a3-d87b2f6e6021" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:48:54.906017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.905991    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6xjq\" (UniqueName: \"kubernetes.io/projected/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-kube-api-access-n6xjq\") pod \"maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:54.906132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.906062    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:54.906224 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.906205    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:54.906336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.906312    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h4q2\" (UniqueName: \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-kube-api-access-5h4q2\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.906386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.906356    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.906386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.906370    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.906386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.906379    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:54.906491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:54.906387    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5377e2b2-cc37-4884-96a0-9a5e2628d968-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:55.007881 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.007806    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:55.008040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.007918    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:55.008040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.007974    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6xjq\" (UniqueName: \"kubernetes.io/projected/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-kube-api-access-n6xjq\") pod \"maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:55.010646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.010614    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:55.010782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.010760    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:48:55.018609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.018584    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:55.020325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.020282    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:55.020475 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.020459    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:48:55.030990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.030969    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6xjq\" (UniqueName: \"kubernetes.io/projected/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-kube-api-access-n6xjq\") pod \"maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:55.095180 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.095152    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:48:55.103107 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.103085    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:55.383804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.383774    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"]
Nov 02 01:48:55.385344 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:48:55.385312    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f028fa5_9865_46ff_b18e_7e738d8c3ef1.slice/crio-8001e55af0c3a07c5de8e43a4d338952df85a6a0d3bcdb223364749f081a7a58 WatchSource:0}: Error finding container 8001e55af0c3a07c5de8e43a4d338952df85a6a0d3bcdb223364749f081a7a58: Status 404 returned error can't find the container with id 8001e55af0c3a07c5de8e43a4d338952df85a6a0d3bcdb223364749f081a7a58
Nov 02 01:48:55.632280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.632201    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"
Nov 02 01:48:55.632280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.632220    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9" event={"ID":"5377e2b2-cc37-4884-96a0-9a5e2628d968","Type":"ContainerDied","Data":"ee66fc30639e44a22230172ecb5e56e104539f93ab075a2163231daffaf40d35"}
Nov 02 01:48:55.632522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.632281    4851 scope.go:122] "RemoveContainer" containerID="6a16f6aa768b57d1737e5f51a12cf40e31d00f16f8faaa7e2ba86c1610b9216c"
Nov 02 01:48:55.634226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.634195    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt" event={"ID":"9f028fa5-9865-46ff-b18e-7e738d8c3ef1","Type":"ContainerStarted","Data":"8001e55af0c3a07c5de8e43a4d338952df85a6a0d3bcdb223364749f081a7a58"}
Nov 02 01:48:55.651714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.651690    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"]
Nov 02 01:48:55.657908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.657883    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3c4ea5-openshift-default-75c759668f-gnvk9"]
Nov 02 01:48:55.841308 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:55.841259    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm" podUID="11a2b0ca-9b50-4b68-b0bb-7d7425bd3655" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:48:56.027375 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:56.027344    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"]
Nov 02 01:48:56.031452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:56.031427    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40220884-76qft"]
Nov 02 01:48:56.643454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:56.643361    4851 generic.go:356] "Generic (PLEG): container finished" podID="9f028fa5-9865-46ff-b18e-7e738d8c3ef1" containerID="1a0704d9a7ec48262526093972d4885c4cc440c688345711ff5c1f5e59a1a398" exitCode=0
Nov 02 01:48:56.643614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:56.643450    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt" event={"ID":"9f028fa5-9865-46ff-b18e-7e738d8c3ef1","Type":"ContainerDied","Data":"1a0704d9a7ec48262526093972d4885c4cc440c688345711ff5c1f5e59a1a398"}
Nov 02 01:48:56.847203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:56.847170    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512964a3-c7ba-4bea-ab47-ab1566234737" path="/var/lib/kubelet/pods/512964a3-c7ba-4bea-ab47-ab1566234737/volumes"
Nov 02 01:48:56.847720 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:56.847638    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" path="/var/lib/kubelet/pods/5377e2b2-cc37-4884-96a0-9a5e2628d968/volumes"
Nov 02 01:48:57.041185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.041150    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"]
Nov 02 01:48:57.047415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.047376    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac77b7df-76sfq"]
Nov 02 01:48:57.596916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.596881    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57" podUID="77229192-c71d-4a53-becc-3338c7de9c81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:48:57.860438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.860368    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:57.948751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.948711    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-openshift-ca\") pod \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") "
Nov 02 01:48:57.948932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.948814    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-xks-ca\") pod \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") "
Nov 02 01:48:57.948932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.948868    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-kube-api-access-n6xjq\" (UniqueName: \"kubernetes.io/projected/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-kube-api-access-n6xjq\") pod \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\" (UID: \"9f028fa5-9865-46ff-b18e-7e738d8c3ef1\") "
Nov 02 01:48:57.949135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.949111    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-openshift-ca" pod "9f028fa5-9865-46ff-b18e-7e738d8c3ef1" (UID: "9f028fa5-9865-46ff-b18e-7e738d8c3ef1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:57.949519 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.949430    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:57.951892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.951871    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-xks-ca" pod "9f028fa5-9865-46ff-b18e-7e738d8c3ef1" (UID: "9f028fa5-9865-46ff-b18e-7e738d8c3ef1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:57.951997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:57.951952    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-kube-api-access-n6xjq" pod "9f028fa5-9865-46ff-b18e-7e738d8c3ef1" (UID: "9f028fa5-9865-46ff-b18e-7e738d8c3ef1"). InnerVolumeSpecName "kube-api-access-n6xjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:58.050472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:58.050436    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:58.050472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:58.050474    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6xjq\" (UniqueName: \"kubernetes.io/projected/9f028fa5-9865-46ff-b18e-7e738d8c3ef1-kube-api-access-n6xjq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:48:58.661774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:58.661744    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"
Nov 02 01:48:58.661774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:58.661764    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt" event={"ID":"9f028fa5-9865-46ff-b18e-7e738d8c3ef1","Type":"ContainerDied","Data":"8001e55af0c3a07c5de8e43a4d338952df85a6a0d3bcdb223364749f081a7a58"}
Nov 02 01:48:58.661994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:58.661836    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8001e55af0c3a07c5de8e43a4d338952df85a6a0d3bcdb223364749f081a7a58"
Nov 02 01:48:58.847785 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:58.847748    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259f5008-f958-49b5-b794-5c525b55040a" path="/var/lib/kubelet/pods/259f5008-f958-49b5-b794-5c525b55040a/volumes"
Nov 02 01:48:59.482899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.482863    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"]
Nov 02 01:48:59.483422 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.483102    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy" containerID="cri-o://eece6565022ab35ef5494332e5255fcf64fe7473f113c96096f7660f7c9242f0" gracePeriod=30
Nov 02 01:48:59.555997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.555962    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"]
Nov 02 01:48:59.579170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.579132    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"]
Nov 02 01:48:59.672910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.672875    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g" podUID="31addff9-dc30-4d8a-8753-8229f834003f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:48:59.995060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.995030    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"]
Nov 02 01:48:59.995643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.995623    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9f028fa5-9865-46ff-b18e-7e738d8c3ef1" containerName="revoke-keys"
Nov 02 01:48:59.995752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.995645    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f028fa5-9865-46ff-b18e-7e738d8c3ef1" containerName="revoke-keys"
Nov 02 01:48:59.995844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.995829    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f028fa5-9865-46ff-b18e-7e738d8c3ef1" containerName="revoke-keys"
Nov 02 01:48:59.995910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.995854    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5377e2b2-cc37-4884-96a0-9a5e2628d968" containerName="istio-proxy"
Nov 02 01:48:59.999581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:48:59.999561    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.004176 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.004153    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:49:00.004356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.004199    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:49:00.008970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.008948    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"]
Nov 02 01:49:00.067628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.067599    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:00.067859 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.067844    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:49:00.074047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.074021    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skf67\" (UniqueName: \"kubernetes.io/projected/54d218ff-a6d0-4d7a-845d-b291076cef06-kube-api-access-skf67\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.074214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.074196    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.074265 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.074229    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.175170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.175139    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.175368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.175178    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.175368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.175257    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skf67\" (UniqueName: \"kubernetes.io/projected/54d218ff-a6d0-4d7a-845d-b291076cef06-kube-api-access-skf67\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.176020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.175939    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.177794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.177763    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.187841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.187799    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skf67\" (UniqueName: \"kubernetes.io/projected/54d218ff-a6d0-4d7a-845d-b291076cef06-kube-api-access-skf67\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.315438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.315352    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:00.645892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.645860    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"]
Nov 02 01:49:00.648833 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:49:00.648801    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d218ff_a6d0_4d7a_845d_b291076cef06.slice/crio-0c246e5d2782d819f2a83777b6eb4c8960e6a0c15c3a37aa9980533778947a8f WatchSource:0}: Error finding container 0c246e5d2782d819f2a83777b6eb4c8960e6a0c15c3a37aa9980533778947a8f: Status 404 returned error can't find the container with id 0c246e5d2782d819f2a83777b6eb4c8960e6a0c15c3a37aa9980533778947a8f
Nov 02 01:49:00.678335 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:00.678264    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5" event={"ID":"54d218ff-a6d0-4d7a-845d-b291076cef06","Type":"ContainerStarted","Data":"0c246e5d2782d819f2a83777b6eb4c8960e6a0c15c3a37aa9980533778947a8f"}
Nov 02 01:49:01.688439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:01.688409    4851 generic.go:356] "Generic (PLEG): container finished" podID="54d218ff-a6d0-4d7a-845d-b291076cef06" containerID="c5ec9a80018c5beba6ff328fec864994986bbd8784fe2465fdd8817cb8621245" exitCode=0
Nov 02 01:49:01.688845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:01.688445    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5" event={"ID":"54d218ff-a6d0-4d7a-845d-b291076cef06","Type":"ContainerDied","Data":"c5ec9a80018c5beba6ff328fec864994986bbd8784fe2465fdd8817cb8621245"}
Nov 02 01:49:02.616985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:02.616952    4851 scope.go:122] "RemoveContainer" containerID="d5b90d616c4649cbe8af42870838afb6b74cf796827bb2341a34cc75a94255ee"
Nov 02 01:49:02.631678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:02.631652    4851 scope.go:122] "RemoveContainer" containerID="ca6dc05eaac96abb73ad41c33b5e4630aaa162b39dbdf384c045808db13f4836"
Nov 02 01:49:02.641993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:02.641975    4851 scope.go:122] "RemoveContainer" containerID="90aa506e7c9b2e8639636359cc4be83abe9b36b95e616cf607e741a3cf281511"
Nov 02 01:49:02.651184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:02.651165    4851 scope.go:122] "RemoveContainer" containerID="cdc536eea8190e9f6782349bc52219be0bd34716485681cc61227df11bb40fb0"
Nov 02 01:49:02.660068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:02.660044    4851 scope.go:122] "RemoveContainer" containerID="926cd98d84691d438a7392fc09ad6cf5a68b6b1a43995a27b38e38384dd65b3a"
Nov 02 01:49:02.669407 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:02.669370    4851 scope.go:122] "RemoveContainer" containerID="38d9605d30f59e6e8f48d4a74960a7fe765e27dfffda01ff2719f44cd478095e"
Nov 02 01:49:02.678216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:02.678194    4851 scope.go:122] "RemoveContainer" containerID="c6d50170ffa54982ee21e9736917e435d663782addc69b7c7de5524fc53195b2"
Nov 02 01:49:02.687117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:02.687098    4851 scope.go:122] "RemoveContainer" containerID="edc4b3f64d500ef7d10dc8bfed212869032c42dca873357a9f0942d7cc98f005"
Nov 02 01:49:02.908392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:02.908368    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:03.030652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.030622    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-xks-ca\") pod \"54d218ff-a6d0-4d7a-845d-b291076cef06\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") "
Nov 02 01:49:03.030885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.030699    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-openshift-ca\") pod \"54d218ff-a6d0-4d7a-845d-b291076cef06\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") "
Nov 02 01:49:03.030885 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.030784    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d218ff-a6d0-4d7a-845d-b291076cef06-kube-api-access-skf67\" (UniqueName: \"kubernetes.io/projected/54d218ff-a6d0-4d7a-845d-b291076cef06-kube-api-access-skf67\") pod \"54d218ff-a6d0-4d7a-845d-b291076cef06\" (UID: \"54d218ff-a6d0-4d7a-845d-b291076cef06\") "
Nov 02 01:49:03.031079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.031054    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-openshift-ca" pod "54d218ff-a6d0-4d7a-845d-b291076cef06" (UID: "54d218ff-a6d0-4d7a-845d-b291076cef06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:03.033031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.033009    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d218ff-a6d0-4d7a-845d-b291076cef06-kube-api-access-skf67" pod "54d218ff-a6d0-4d7a-845d-b291076cef06" (UID: "54d218ff-a6d0-4d7a-845d-b291076cef06"). InnerVolumeSpecName "kube-api-access-skf67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:03.033110 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.033068    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-xks-ca" pod "54d218ff-a6d0-4d7a-845d-b291076cef06" (UID: "54d218ff-a6d0-4d7a-845d-b291076cef06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:03.132331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.132239    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:03.132331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.132265    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d218ff-a6d0-4d7a-845d-b291076cef06-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:03.132331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.132277    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skf67\" (UniqueName: \"kubernetes.io/projected/54d218ff-a6d0-4d7a-845d-b291076cef06-kube-api-access-skf67\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:03.712038 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.712012    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"
Nov 02 01:49:03.712225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.712051    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5" event={"ID":"54d218ff-a6d0-4d7a-845d-b291076cef06","Type":"ContainerDied","Data":"0c246e5d2782d819f2a83777b6eb4c8960e6a0c15c3a37aa9980533778947a8f"}
Nov 02 01:49:03.712225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:03.712095    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c246e5d2782d819f2a83777b6eb4c8960e6a0c15c3a37aa9980533778947a8f"
Nov 02 01:49:04.724971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.724934    4851 generic.go:356] "Generic (PLEG): container finished" podID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerID="eece6565022ab35ef5494332e5255fcf64fe7473f113c96096f7660f7c9242f0" exitCode=0
Nov 02 01:49:04.725468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.725003    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" event={"ID":"fa8656c6-8347-438b-b2ee-b2ae03c69590","Type":"ContainerDied","Data":"eece6565022ab35ef5494332e5255fcf64fe7473f113c96096f7660f7c9242f0"}
Nov 02 01:49:04.725468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.725075    4851 scope.go:122] "RemoveContainer" containerID="0511219424eac1bed532c7b2fe2bc593937ff78743c479f5b01dfe7d6a1a728a"
Nov 02 01:49:04.775496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.775474    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:49:04.837687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.837638    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq" podUID="49ae4474-4f10-47b8-86a3-d87b2f6e6021" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:04.880441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.880401    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-credential-socket\") pod \"fa8656c6-8347-438b-b2ee-b2ae03c69590\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") "
Nov 02 01:49:04.880650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.880504    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-kube-api-access-nlmpl\" (UniqueName: \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-kube-api-access-nlmpl\") pod \"fa8656c6-8347-438b-b2ee-b2ae03c69590\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") "
Nov 02 01:49:04.880650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.880536    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-data\") pod \"fa8656c6-8347-438b-b2ee-b2ae03c69590\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") "
Nov 02 01:49:04.880650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.880605    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-envoy\") pod \"fa8656c6-8347-438b-b2ee-b2ae03c69590\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") "
Nov 02 01:49:04.880826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.880651    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-podinfo\") pod \"fa8656c6-8347-438b-b2ee-b2ae03c69590\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") "
Nov 02 01:49:04.880826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.880682    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-token\" (UniqueName: \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-token\") pod \"fa8656c6-8347-438b-b2ee-b2ae03c69590\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") "
Nov 02 01:49:04.880826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.880744    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-certs\") pod \"fa8656c6-8347-438b-b2ee-b2ae03c69590\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") "
Nov 02 01:49:04.880826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.880795    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-socket\") pod \"fa8656c6-8347-438b-b2ee-b2ae03c69590\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") "
Nov 02 01:49:04.881054 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.880843    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8656c6-8347-438b-b2ee-b2ae03c69590-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8656c6-8347-438b-b2ee-b2ae03c69590-istiod-ca-cert\") pod \"fa8656c6-8347-438b-b2ee-b2ae03c69590\" (UID: \"fa8656c6-8347-438b-b2ee-b2ae03c69590\") "
Nov 02 01:49:04.881764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.881734    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8656c6-8347-438b-b2ee-b2ae03c69590-istiod-ca-cert" pod "fa8656c6-8347-438b-b2ee-b2ae03c69590" (UID: "fa8656c6-8347-438b-b2ee-b2ae03c69590"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:04.882152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.882126    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-credential-socket" pod "fa8656c6-8347-438b-b2ee-b2ae03c69590" (UID: "fa8656c6-8347-438b-b2ee-b2ae03c69590"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:04.883650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.883162    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-socket" pod "fa8656c6-8347-438b-b2ee-b2ae03c69590" (UID: "fa8656c6-8347-438b-b2ee-b2ae03c69590"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:04.883650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.883359    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-data" pod "fa8656c6-8347-438b-b2ee-b2ae03c69590" (UID: "fa8656c6-8347-438b-b2ee-b2ae03c69590"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:04.883650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.883429    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-certs" pod "fa8656c6-8347-438b-b2ee-b2ae03c69590" (UID: "fa8656c6-8347-438b-b2ee-b2ae03c69590"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:04.898525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.890564    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-kube-api-access-nlmpl" pod "fa8656c6-8347-438b-b2ee-b2ae03c69590" (UID: "fa8656c6-8347-438b-b2ee-b2ae03c69590"). InnerVolumeSpecName "kube-api-access-nlmpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:04.898525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.896739    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-podinfo" pod "fa8656c6-8347-438b-b2ee-b2ae03c69590" (UID: "fa8656c6-8347-438b-b2ee-b2ae03c69590"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:49:04.898525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.896859    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-envoy" pod "fa8656c6-8347-438b-b2ee-b2ae03c69590" (UID: "fa8656c6-8347-438b-b2ee-b2ae03c69590"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:04.899441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.899412    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-token" pod "fa8656c6-8347-438b-b2ee-b2ae03c69590" (UID: "fa8656c6-8347-438b-b2ee-b2ae03c69590"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:04.983052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.983019    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:04.983052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.983050    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlmpl\" (UniqueName: \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-kube-api-access-nlmpl\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:04.983052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.983067    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:04.983324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.983080    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:04.983324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.983106    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:04.983324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.983116    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8656c6-8347-438b-b2ee-b2ae03c69590-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:04.983324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.983124    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:04.983324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.983133    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8656c6-8347-438b-b2ee-b2ae03c69590-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:04.983324 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.983141    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8656c6-8347-438b-b2ee-b2ae03c69590-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:04.997106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:04.997057    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"]
Nov 02 01:49:05.000112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:05.000088    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d4a0-w24p2"]
Nov 02 01:49:05.166317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:05.156524    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"]
Nov 02 01:49:05.733548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:05.733516    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"
Nov 02 01:49:05.734022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:05.733551    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx" event={"ID":"fa8656c6-8347-438b-b2ee-b2ae03c69590","Type":"ContainerDied","Data":"ea156564013fd572d4835e2fb4ccc354570c7db7382dab84214f40f2bb6efa67"}
Nov 02 01:49:05.734022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:05.733609    4851 scope.go:122] "RemoveContainer" containerID="eece6565022ab35ef5494332e5255fcf64fe7473f113c96096f7660f7c9242f0"
Nov 02 01:49:05.762187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:05.762154    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"]
Nov 02 01:49:05.765027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:05.765000    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-lt6vx"]
Nov 02 01:49:05.841527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:05.841489    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm" podUID="11a2b0ca-9b50-4b68-b0bb-7d7425bd3655" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:06.493175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.493149    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:49:06.516584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.516560    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:49:06.604758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.604727    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ae4474-4f10-47b8-86a3-d87b2f6e6021-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ae4474-4f10-47b8-86a3-d87b2f6e6021-plugins-config-volume\") pod \"49ae4474-4f10-47b8-86a3-d87b2f6e6021\" (UID: \"49ae4474-4f10-47b8-86a3-d87b2f6e6021\") "
Nov 02 01:49:06.604968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.604806    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ae4474-4f10-47b8-86a3-d87b2f6e6021-kube-api-access-9tg6g\" (UniqueName: \"kubernetes.io/projected/49ae4474-4f10-47b8-86a3-d87b2f6e6021-kube-api-access-9tg6g\") pod \"49ae4474-4f10-47b8-86a3-d87b2f6e6021\" (UID: \"49ae4474-4f10-47b8-86a3-d87b2f6e6021\") "
Nov 02 01:49:06.604968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.604867    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-kube-api-access-5h92q\" (UniqueName: \"kubernetes.io/projected/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-kube-api-access-5h92q\") pod \"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655\" (UID: \"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655\") "
Nov 02 01:49:06.605103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.604986    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-plugins-config-volume\") pod \"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655\" (UID: \"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655\") "
Nov 02 01:49:06.605364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.605284    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ae4474-4f10-47b8-86a3-d87b2f6e6021-plugins-config-volume" pod "49ae4474-4f10-47b8-86a3-d87b2f6e6021" (UID: "49ae4474-4f10-47b8-86a3-d87b2f6e6021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:06.605364 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.605353    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-plugins-config-volume" pod "11a2b0ca-9b50-4b68-b0bb-7d7425bd3655" (UID: "11a2b0ca-9b50-4b68-b0bb-7d7425bd3655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:06.605537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.605499    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:06.605537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.605520    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ae4474-4f10-47b8-86a3-d87b2f6e6021-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:06.607183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.607164    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ae4474-4f10-47b8-86a3-d87b2f6e6021-kube-api-access-9tg6g" pod "49ae4474-4f10-47b8-86a3-d87b2f6e6021" (UID: "49ae4474-4f10-47b8-86a3-d87b2f6e6021"). InnerVolumeSpecName "kube-api-access-9tg6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:06.607255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.607242    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-kube-api-access-5h92q" pod "11a2b0ca-9b50-4b68-b0bb-7d7425bd3655" (UID: "11a2b0ca-9b50-4b68-b0bb-7d7425bd3655"). InnerVolumeSpecName "kube-api-access-5h92q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:06.706525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.706445    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tg6g\" (UniqueName: \"kubernetes.io/projected/49ae4474-4f10-47b8-86a3-d87b2f6e6021-kube-api-access-9tg6g\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:06.706525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.706474    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h92q\" (UniqueName: \"kubernetes.io/projected/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655-kube-api-access-5h92q\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:06.743412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.743374    4851 generic.go:356] "Generic (PLEG): container finished" podID="49ae4474-4f10-47b8-86a3-d87b2f6e6021" containerID="c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef" exitCode=0
Nov 02 01:49:06.743861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.743437    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"
Nov 02 01:49:06.743861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.743455    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq" event={"ID":"49ae4474-4f10-47b8-86a3-d87b2f6e6021","Type":"ContainerDied","Data":"c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef"}
Nov 02 01:49:06.743861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.743500    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq" event={"ID":"49ae4474-4f10-47b8-86a3-d87b2f6e6021","Type":"ContainerDied","Data":"e8b90d47c151794823be869769ce846598c98c430d39f395f24a7f4da4856989"}
Nov 02 01:49:06.743861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.743530    4851 scope.go:122] "RemoveContainer" containerID="c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef"
Nov 02 01:49:06.745166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.745146    4851 generic.go:356] "Generic (PLEG): container finished" podID="11a2b0ca-9b50-4b68-b0bb-7d7425bd3655" containerID="a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c" exitCode=0
Nov 02 01:49:06.745237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.745211    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"
Nov 02 01:49:06.745284 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.745231    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm" event={"ID":"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655","Type":"ContainerDied","Data":"a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c"}
Nov 02 01:49:06.745284 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.745274    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm" event={"ID":"11a2b0ca-9b50-4b68-b0bb-7d7425bd3655","Type":"ContainerDied","Data":"724709d1d37caf0fa31958b9446a70941e22a7cfd83cfa6549509b73195d2c10"}
Nov 02 01:49:06.754581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.754398    4851 scope.go:122] "RemoveContainer" containerID="c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef"
Nov 02 01:49:06.754755 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:06.754731    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef\": container with ID starting with c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef not found: ID does not exist" containerID="c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef"
Nov 02 01:49:06.754824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.754763    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef"} err="failed to get container status \"c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef\": rpc error: code = NotFound desc = could not find container \"c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef\": container with ID starting with c387cb5ec20ce99f0d9b0df557d939fe23595e9be83924b73067690e289815ef not found: ID does not exist"
Nov 02 01:49:06.754824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.754782    4851 scope.go:122] "RemoveContainer" containerID="a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c"
Nov 02 01:49:06.764635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.764616    4851 scope.go:122] "RemoveContainer" containerID="a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c"
Nov 02 01:49:06.764869 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:06.764850    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c\": container with ID starting with a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c not found: ID does not exist" containerID="a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c"
Nov 02 01:49:06.764934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.764875    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c"} err="failed to get container status \"a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c\": rpc error: code = NotFound desc = could not find container \"a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c\": container with ID starting with a707cec81925c9ea6b99c0e6cb725b601e1d2f0e87a919539fce4227241a864c not found: ID does not exist"
Nov 02 01:49:06.775222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.775200    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"]
Nov 02 01:49:06.779754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.779735    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3c4ea5-64c7468f5-6rcjq"]
Nov 02 01:49:06.811403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.808086    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"]
Nov 02 01:49:06.815315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.812228    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3c4ea5-777c64c859-mfgrm"]
Nov 02 01:49:06.877330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.876685    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a2b0ca-9b50-4b68-b0bb-7d7425bd3655" path="/var/lib/kubelet/pods/11a2b0ca-9b50-4b68-b0bb-7d7425bd3655/volumes"
Nov 02 01:49:06.877330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.877136    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ae4474-4f10-47b8-86a3-d87b2f6e6021" path="/var/lib/kubelet/pods/49ae4474-4f10-47b8-86a3-d87b2f6e6021/volumes"
Nov 02 01:49:06.877882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.877853    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e6c7c4-5f87-4295-91f2-7f0a4795bc06" path="/var/lib/kubelet/pods/70e6c7c4-5f87-4295-91f2-7f0a4795bc06/volumes"
Nov 02 01:49:06.878682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:06.878661    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" path="/var/lib/kubelet/pods/fa8656c6-8347-438b-b2ee-b2ae03c69590/volumes"
Nov 02 01:49:07.596727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:07.596640    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57" podUID="77229192-c71d-4a53-becc-3338c7de9c81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:08.646938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.646908    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"]
Nov 02 01:49:08.647435 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647422    4851 cpu_manager.go:414] "containerMap: removing container" podUID="54d218ff-a6d0-4d7a-845d-b291076cef06" containerName="revoke-keys"
Nov 02 01:49:08.647491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647436    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d218ff-a6d0-4d7a-845d-b291076cef06" containerName="revoke-keys"
Nov 02 01:49:08.647491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647451    4851 cpu_manager.go:414] "containerMap: removing container" podUID="49ae4474-4f10-47b8-86a3-d87b2f6e6021" containerName="payload-pre-processing"
Nov 02 01:49:08.647491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647456    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ae4474-4f10-47b8-86a3-d87b2f6e6021" containerName="payload-pre-processing"
Nov 02 01:49:08.647491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647476    4851 cpu_manager.go:414] "containerMap: removing container" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy"
Nov 02 01:49:08.647491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647481    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy"
Nov 02 01:49:08.647673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647505    4851 cpu_manager.go:414] "containerMap: removing container" podUID="11a2b0ca-9b50-4b68-b0bb-7d7425bd3655" containerName="payload-processing"
Nov 02 01:49:08.647673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647509    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a2b0ca-9b50-4b68-b0bb-7d7425bd3655" containerName="payload-processing"
Nov 02 01:49:08.647673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647577    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d218ff-a6d0-4d7a-845d-b291076cef06" containerName="revoke-keys"
Nov 02 01:49:08.647673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647587    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ae4474-4f10-47b8-86a3-d87b2f6e6021" containerName="payload-pre-processing"
Nov 02 01:49:08.647673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647595    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy"
Nov 02 01:49:08.647673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647605    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy"
Nov 02 01:49:08.647673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.647611    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a2b0ca-9b50-4b68-b0bb-7d7425bd3655" containerName="payload-processing"
Nov 02 01:49:08.652695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.652678    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.656756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.656736    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:08.656756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.656752    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:49:08.657747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.657726    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:49:08.657858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.657844    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:49:08.662353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.662333    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"]
Nov 02 01:49:08.669131 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.669107    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"]
Nov 02 01:49:08.669622 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.669608    4851 cpu_manager.go:414] "containerMap: removing container" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy"
Nov 02 01:49:08.669679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.669624    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8656c6-8347-438b-b2ee-b2ae03c69590" containerName="istio-proxy"
Nov 02 01:49:08.673646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.673625    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:08.682910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.682881    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"]
Nov 02 01:49:08.829067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.829037    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4c104cf-7l4hk\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:08.829274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.829075    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdwhq\" (UniqueName: \"kubernetes.io/projected/131cb34c-381c-4fdf-85fa-1ca7609a16dd-kube-api-access-vdwhq\") pod \"maas-api-revoke-sub-d9f59528-bln6z\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.829274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.829152    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbfd5\" (UniqueName: \"kubernetes.io/projected/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-kube-api-access-bbfd5\") pod \"maas-api-revoke-sub-e4c104cf-7l4hk\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:08.829274 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.829261    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4c104cf-7l4hk\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:08.829520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.829277    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9f59528-bln6z\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.829520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.829406    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9f59528-bln6z\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.931097 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.931067    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4c104cf-7l4hk\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:08.931097 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.931097    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9f59528-bln6z\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.931455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.931157    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9f59528-bln6z\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.931455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.931213    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4c104cf-7l4hk\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:08.931455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.931249    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdwhq\" (UniqueName: \"kubernetes.io/projected/131cb34c-381c-4fdf-85fa-1ca7609a16dd-kube-api-access-vdwhq\") pod \"maas-api-revoke-sub-d9f59528-bln6z\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.931455 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.931327    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbfd5\" (UniqueName: \"kubernetes.io/projected/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-kube-api-access-bbfd5\") pod \"maas-api-revoke-sub-e4c104cf-7l4hk\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:08.931851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.931826    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4c104cf-7l4hk\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:08.931967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.931944    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9f59528-bln6z\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.934125 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.934104    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9f59528-bln6z\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.934419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.934397    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4c104cf-7l4hk\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:08.943527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.943504    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbfd5\" (UniqueName: \"kubernetes.io/projected/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-kube-api-access-bbfd5\") pod \"maas-api-revoke-sub-e4c104cf-7l4hk\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:08.943834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.943615    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdwhq\" (UniqueName: \"kubernetes.io/projected/131cb34c-381c-4fdf-85fa-1ca7609a16dd-kube-api-access-vdwhq\") pod \"maas-api-revoke-sub-d9f59528-bln6z\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.971178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.971156    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:08.992347 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:08.992321    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:09.294449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:09.294420    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"]
Nov 02 01:49:09.320888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:09.320859    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"]
Nov 02 01:49:09.329594 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:49:09.329569    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0bf252_75d1_4bbf_a907_f4e09c2bd0bd.slice/crio-a8f4b100d8b34b9108c7e68589e856250412ed521f7dfef3b40b60648379f5e4 WatchSource:0}: Error finding container a8f4b100d8b34b9108c7e68589e856250412ed521f7dfef3b40b60648379f5e4: Status 404 returned error can't find the container with id a8f4b100d8b34b9108c7e68589e856250412ed521f7dfef3b40b60648379f5e4
Nov 02 01:49:09.672342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:09.672311    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g" podUID="31addff9-dc30-4d8a-8753-8229f834003f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:09.797203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:09.797164    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z" event={"ID":"131cb34c-381c-4fdf-85fa-1ca7609a16dd","Type":"ContainerStarted","Data":"60744a142430dd8d9841efa4de2a122900d7be378bb48b33a63eeed86fbd4aba"}
Nov 02 01:49:09.799534 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:09.799500    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk" event={"ID":"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd","Type":"ContainerStarted","Data":"a8f4b100d8b34b9108c7e68589e856250412ed521f7dfef3b40b60648379f5e4"}
Nov 02 01:49:10.117220 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:10.117185    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"]
Nov 02 01:49:10.182434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:10.182397    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw" podUID="1c6fddd1-829f-4aef-88d8-86ecab8099c6" containerName="payload-processing" containerID="cri-o://f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772" gracePeriod=25
Nov 02 01:49:10.808638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:10.808545    4851 generic.go:356] "Generic (PLEG): container finished" podID="131cb34c-381c-4fdf-85fa-1ca7609a16dd" containerID="b0f95de92fafd49450a769e49c483225669c27acf26c815b89929e216e91854f" exitCode=0
Nov 02 01:49:10.809027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:10.808633    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z" event={"ID":"131cb34c-381c-4fdf-85fa-1ca7609a16dd","Type":"ContainerDied","Data":"b0f95de92fafd49450a769e49c483225669c27acf26c815b89929e216e91854f"}
Nov 02 01:49:10.810366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:10.810341    4851 generic.go:356] "Generic (PLEG): container finished" podID="2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd" containerID="6588dc74a0ec0099b05c47c522f20ab54eea07b3893c732a928b15ff38d05d68" exitCode=0
Nov 02 01:49:10.810499 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:10.810424    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk" event={"ID":"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd","Type":"ContainerDied","Data":"6588dc74a0ec0099b05c47c522f20ab54eea07b3893c732a928b15ff38d05d68"}
Nov 02 01:49:11.022316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:11.022253    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"]
Nov 02 01:49:11.026438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:11.026413    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95671cc9-mdxbs"]
Nov 02 01:49:11.978067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:11.978034    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:12.019827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.019802    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:12.088566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.088486    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-openshift-ca\") pod \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") "
Nov 02 01:49:12.088566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.088529    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131cb34c-381c-4fdf-85fa-1ca7609a16dd-kube-api-access-vdwhq\" (UniqueName: \"kubernetes.io/projected/131cb34c-381c-4fdf-85fa-1ca7609a16dd-kube-api-access-vdwhq\") pod \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") "
Nov 02 01:49:12.088754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.088636    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-xks-ca\") pod \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\" (UID: \"131cb34c-381c-4fdf-85fa-1ca7609a16dd\") "
Nov 02 01:49:12.088874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.088852    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-openshift-ca" pod "131cb34c-381c-4fdf-85fa-1ca7609a16dd" (UID: "131cb34c-381c-4fdf-85fa-1ca7609a16dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:12.089080 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.089051    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:12.090888 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.090866    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131cb34c-381c-4fdf-85fa-1ca7609a16dd-kube-api-access-vdwhq" pod "131cb34c-381c-4fdf-85fa-1ca7609a16dd" (UID: "131cb34c-381c-4fdf-85fa-1ca7609a16dd"). InnerVolumeSpecName "kube-api-access-vdwhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:12.090983 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.090964    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-xks-ca" pod "131cb34c-381c-4fdf-85fa-1ca7609a16dd" (UID: "131cb34c-381c-4fdf-85fa-1ca7609a16dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:12.189856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.189825    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-openshift-ca\") pod \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") "
Nov 02 01:49:12.190045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.189899    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-xks-ca\") pod \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") "
Nov 02 01:49:12.190045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.189983    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-kube-api-access-bbfd5\" (UniqueName: \"kubernetes.io/projected/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-kube-api-access-bbfd5\") pod \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\" (UID: \"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd\") "
Nov 02 01:49:12.190212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.190198    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/131cb34c-381c-4fdf-85fa-1ca7609a16dd-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:12.190277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.190213    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdwhq\" (UniqueName: \"kubernetes.io/projected/131cb34c-381c-4fdf-85fa-1ca7609a16dd-kube-api-access-vdwhq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:12.190498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.190370    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-openshift-ca" pod "2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd" (UID: "2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:12.192636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.192605    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-xks-ca" pod "2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd" (UID: "2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:12.193994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.193973    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-kube-api-access-bbfd5" pod "2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd" (UID: "2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd"). InnerVolumeSpecName "kube-api-access-bbfd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:12.298481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.292924    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:12.298481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.292968    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbfd5\" (UniqueName: \"kubernetes.io/projected/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-kube-api-access-bbfd5\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:12.298481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.292984    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:12.377217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.377119    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"]
Nov 02 01:49:12.377462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.377364    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerName="istio-proxy" containerID="cri-o://dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad" gracePeriod=30
Nov 02 01:49:12.836157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.836110    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z" event={"ID":"131cb34c-381c-4fdf-85fa-1ca7609a16dd","Type":"ContainerDied","Data":"60744a142430dd8d9841efa4de2a122900d7be378bb48b33a63eeed86fbd4aba"}
Nov 02 01:49:12.836386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.836203    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60744a142430dd8d9841efa4de2a122900d7be378bb48b33a63eeed86fbd4aba"
Nov 02 01:49:12.836386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.836124    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"
Nov 02 01:49:12.837721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.837698    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"
Nov 02 01:49:12.837856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.837726    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk" event={"ID":"2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd","Type":"ContainerDied","Data":"a8f4b100d8b34b9108c7e68589e856250412ed521f7dfef3b40b60648379f5e4"}
Nov 02 01:49:12.837856 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.837770    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f4b100d8b34b9108c7e68589e856250412ed521f7dfef3b40b60648379f5e4"
Nov 02 01:49:12.853522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:12.853492    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd00767b-a324-4d8c-96a5-30fee3f05008" path="/var/lib/kubelet/pods/bd00767b-a324-4d8c-96a5-30fee3f05008/volumes"
Nov 02 01:49:13.438225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.438189    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"]
Nov 02 01:49:13.442924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.442892    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0f32d6-882f5"]
Nov 02 01:49:13.484688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.484650    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"]
Nov 02 01:49:13.485240 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.485221    4851 cpu_manager.go:414] "containerMap: removing container" podUID="2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd" containerName="revoke-keys"
Nov 02 01:49:13.485378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.485241    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd" containerName="revoke-keys"
Nov 02 01:49:13.485378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.485275    4851 cpu_manager.go:414] "containerMap: removing container" podUID="131cb34c-381c-4fdf-85fa-1ca7609a16dd" containerName="revoke-keys"
Nov 02 01:49:13.485378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.485284    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131cb34c-381c-4fdf-85fa-1ca7609a16dd" containerName="revoke-keys"
Nov 02 01:49:13.485543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.485446    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd" containerName="revoke-keys"
Nov 02 01:49:13.485543 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.485464    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="131cb34c-381c-4fdf-85fa-1ca7609a16dd" containerName="revoke-keys"
Nov 02 01:49:13.490802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.490777    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.495972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.495504    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:49:13.495972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.495697    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:49:13.495972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.495739    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:13.495972 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.495784    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:49:13.498483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.498462    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"]
Nov 02 01:49:13.535724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.535189    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.535724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.535386    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jts6l\" (UniqueName: \"kubernetes.io/projected/4c69e684-14de-4c9a-9166-3e4cacc890bc-kube-api-access-jts6l\") pod \"maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.535724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.535489    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.637135 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.637102    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.637381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.637178    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.637381 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.637340    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jts6l\" (UniqueName: \"kubernetes.io/projected/4c69e684-14de-4c9a-9166-3e4cacc890bc-kube-api-access-jts6l\") pod \"maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.637799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.637777    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.639651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.639627    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.651683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.651657    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jts6l\" (UniqueName: \"kubernetes.io/projected/4c69e684-14de-4c9a-9166-3e4cacc890bc-kube-api-access-jts6l\") pod \"maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.763433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.763397    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:49:13.808167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.808141    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:49:13.839786 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.839750    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31addff9-dc30-4d8a-8753-8229f834003f-kube-api-access-j7crc\" (UniqueName: \"kubernetes.io/projected/31addff9-dc30-4d8a-8753-8229f834003f-kube-api-access-j7crc\") pod \"31addff9-dc30-4d8a-8753-8229f834003f\" (UID: \"31addff9-dc30-4d8a-8753-8229f834003f\") "
Nov 02 01:49:13.839963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.839884    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31addff9-dc30-4d8a-8753-8229f834003f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31addff9-dc30-4d8a-8753-8229f834003f-plugins-config-volume\") pod \"31addff9-dc30-4d8a-8753-8229f834003f\" (UID: \"31addff9-dc30-4d8a-8753-8229f834003f\") "
Nov 02 01:49:13.839963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.839926    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77229192-c71d-4a53-becc-3338c7de9c81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77229192-c71d-4a53-becc-3338c7de9c81-plugins-config-volume\") pod \"77229192-c71d-4a53-becc-3338c7de9c81\" (UID: \"77229192-c71d-4a53-becc-3338c7de9c81\") "
Nov 02 01:49:13.840096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.840001    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77229192-c71d-4a53-becc-3338c7de9c81-kube-api-access-7x6q4\" (UniqueName: \"kubernetes.io/projected/77229192-c71d-4a53-becc-3338c7de9c81-kube-api-access-7x6q4\") pod \"77229192-c71d-4a53-becc-3338c7de9c81\" (UID: \"77229192-c71d-4a53-becc-3338c7de9c81\") "
Nov 02 01:49:13.841701 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.841666    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31addff9-dc30-4d8a-8753-8229f834003f-plugins-config-volume" pod "31addff9-dc30-4d8a-8753-8229f834003f" (UID: "31addff9-dc30-4d8a-8753-8229f834003f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:13.841847 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.841782    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77229192-c71d-4a53-becc-3338c7de9c81-plugins-config-volume" pod "77229192-c71d-4a53-becc-3338c7de9c81" (UID: "77229192-c71d-4a53-becc-3338c7de9c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:13.842944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.842910    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77229192-c71d-4a53-becc-3338c7de9c81-kube-api-access-7x6q4" pod "77229192-c71d-4a53-becc-3338c7de9c81" (UID: "77229192-c71d-4a53-becc-3338c7de9c81"). InnerVolumeSpecName "kube-api-access-7x6q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:13.843562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.843533    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31addff9-dc30-4d8a-8753-8229f834003f-kube-api-access-j7crc" pod "31addff9-dc30-4d8a-8753-8229f834003f" (UID: "31addff9-dc30-4d8a-8753-8229f834003f"). InnerVolumeSpecName "kube-api-access-j7crc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:13.848034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.848003    4851 generic.go:356] "Generic (PLEG): container finished" podID="31addff9-dc30-4d8a-8753-8229f834003f" containerID="e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65" exitCode=0
Nov 02 01:49:13.848155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.848095    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"
Nov 02 01:49:13.848230 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.848140    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g" event={"ID":"31addff9-dc30-4d8a-8753-8229f834003f","Type":"ContainerDied","Data":"e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65"}
Nov 02 01:49:13.848230 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.848201    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g" event={"ID":"31addff9-dc30-4d8a-8753-8229f834003f","Type":"ContainerDied","Data":"537cd0fd52ef78edf374d8191c363998cbfa4e7b17c3240a75f44ca7e7408448"}
Nov 02 01:49:13.848383 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.848232    4851 scope.go:122] "RemoveContainer" containerID="e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65"
Nov 02 01:49:13.850117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.850089    4851 generic.go:356] "Generic (PLEG): container finished" podID="77229192-c71d-4a53-becc-3338c7de9c81" containerID="a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b" exitCode=0
Nov 02 01:49:13.850241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.850134    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"
Nov 02 01:49:13.850241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.850156    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57" event={"ID":"77229192-c71d-4a53-becc-3338c7de9c81","Type":"ContainerDied","Data":"a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b"}
Nov 02 01:49:13.850241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.850197    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57" event={"ID":"77229192-c71d-4a53-becc-3338c7de9c81","Type":"ContainerDied","Data":"a1bf7ae02431693f1bc1489f5add333cc72610971200d52ab8c48742c505cbab"}
Nov 02 01:49:13.861607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.861585    4851 scope.go:122] "RemoveContainer" containerID="e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65"
Nov 02 01:49:13.861959 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:13.861916    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65\": container with ID starting with e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65 not found: ID does not exist" containerID="e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65"
Nov 02 01:49:13.862073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.861953    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65"} err="failed to get container status \"e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65\": rpc error: code = NotFound desc = could not find container \"e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65\": container with ID starting with e3dba789c3146dd87eb1c653277c4dddb384fbbf2065dda1bc29f2e19b2f9d65 not found: ID does not exist"
Nov 02 01:49:13.862073 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.861977    4851 scope.go:122] "RemoveContainer" containerID="a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b"
Nov 02 01:49:13.875601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.875580    4851 scope.go:122] "RemoveContainer" containerID="a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b"
Nov 02 01:49:13.875909 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:13.875889    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b\": container with ID starting with a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b not found: ID does not exist" containerID="a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b"
Nov 02 01:49:13.875981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.875916    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b"} err="failed to get container status \"a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b\": rpc error: code = NotFound desc = could not find container \"a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b\": container with ID starting with a01af70b1002ae70142b173e40327b93ec6ebafbc5c315852da297efefe3414b not found: ID does not exist"
Nov 02 01:49:13.877494 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.877468    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"]
Nov 02 01:49:13.881133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.881110    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-eaec8c-646c45d8d5-gxp9g"]
Nov 02 01:49:13.892912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.892889    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"]
Nov 02 01:49:13.897183 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.897165    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-eaec8c-9f8b6f675-hvq57"]
Nov 02 01:49:13.924356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.924329    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:13.942103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.942075    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31addff9-dc30-4d8a-8753-8229f834003f-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:13.942213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.942104    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77229192-c71d-4a53-becc-3338c7de9c81-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:13.942213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.942120    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x6q4\" (UniqueName: \"kubernetes.io/projected/77229192-c71d-4a53-becc-3338c7de9c81-kube-api-access-7x6q4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:13.942213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:13.942135    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7crc\" (UniqueName: \"kubernetes.io/projected/31addff9-dc30-4d8a-8753-8229f834003f-kube-api-access-j7crc\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:14.209409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.209379    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"]
Nov 02 01:49:14.211781 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:49:14.211744    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c69e684_14de_4c9a_9166_3e4cacc890bc.slice/crio-e8e212eea30babfcd9378695687fbae89bcfcaaa2af909dc55035effa5fb912c WatchSource:0}: Error finding container e8e212eea30babfcd9378695687fbae89bcfcaaa2af909dc55035effa5fb912c: Status 404 returned error can't find the container with id e8e212eea30babfcd9378695687fbae89bcfcaaa2af909dc55035effa5fb912c
Nov 02 01:49:14.596580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.596540    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main" containerID="cri-o://85feb868483c9a8f77a74a7c6028b558a75ffeb098f7033fba35983ce5e7ca7a" gracePeriod=45
Nov 02 01:49:14.600859 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.600818    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" containerID="cri-o://cc4a2c0c9fe2d30255ea7bd57d6186a63b55c9ddd70f9c6d8de895f7a3e85510" gracePeriod=45
Nov 02 01:49:14.768409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.768363    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused"
Nov 02 01:49:14.862348 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.862250    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31addff9-dc30-4d8a-8753-8229f834003f" path="/var/lib/kubelet/pods/31addff9-dc30-4d8a-8753-8229f834003f/volumes"
Nov 02 01:49:14.863388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.863363    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d218ff-a6d0-4d7a-845d-b291076cef06" path="/var/lib/kubelet/pods/54d218ff-a6d0-4d7a-845d-b291076cef06/volumes"
Nov 02 01:49:14.863974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.863953    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77229192-c71d-4a53-becc-3338c7de9c81" path="/var/lib/kubelet/pods/77229192-c71d-4a53-becc-3338c7de9c81/volumes"
Nov 02 01:49:14.886929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.886901    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc" event={"ID":"4c69e684-14de-4c9a-9166-3e4cacc890bc","Type":"ContainerStarted","Data":"d146bc6e9ddb9e9ef8a3faf62eb0068e35ee87fd9bccbb709112ca5b4ef89c03"}
Nov 02 01:49:14.887148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.887130    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc" event={"ID":"4c69e684-14de-4c9a-9166-3e4cacc890bc","Type":"ContainerStarted","Data":"e8e212eea30babfcd9378695687fbae89bcfcaaa2af909dc55035effa5fb912c"}
Nov 02 01:49:14.889365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.889338    4851 generic.go:356] "Generic (PLEG): container finished" podID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerID="cc4a2c0c9fe2d30255ea7bd57d6186a63b55c9ddd70f9c6d8de895f7a3e85510" exitCode=0
Nov 02 01:49:14.889742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.889718    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" event={"ID":"a0126455-bcc3-4334-a73c-e53b3db5bce9","Type":"ContainerDied","Data":"cc4a2c0c9fe2d30255ea7bd57d6186a63b55c9ddd70f9c6d8de895f7a3e85510"}
Nov 02 01:49:14.895044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.894987    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:49:14.898707 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.898683    4851 generic.go:356] "Generic (PLEG): container finished" podID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerID="85feb868483c9a8f77a74a7c6028b558a75ffeb098f7033fba35983ce5e7ca7a" exitCode=0
Nov 02 01:49:14.898929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.898909    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" event={"ID":"e54d3a0c-8652-4445-b6e6-52d670571cfe","Type":"ContainerDied","Data":"85feb868483c9a8f77a74a7c6028b558a75ffeb098f7033fba35983ce5e7ca7a"}
Nov 02 01:49:14.911425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.911383    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc" podStartSLOduration=1.911370066 podStartE2EDuration="1.911370066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:14.907871307 +0000 UTC m=+2060.212804137" watchObservedRunningTime="2026-11-02 01:49:14.911370066 +0000 UTC m=+2060.216302891"
Nov 02 01:49:14.912993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.912970    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.952519    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54d3a0c-8652-4445-b6e6-52d670571cfe-tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d3a0c-8652-4445-b6e6-52d670571cfe-tls-certs\") pod \"e54d3a0c-8652-4445-b6e6-52d670571cfe\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.952593    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-home\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-home\") pod \"e54d3a0c-8652-4445-b6e6-52d670571cfe\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.952660    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-model-cache\") pod \"a0126455-bcc3-4334-a73c-e53b3db5bce9\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.952746    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-home\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-home\") pod \"a0126455-bcc3-4334-a73c-e53b3db5bce9\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.952801    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0126455-bcc3-4334-a73c-e53b3db5bce9-kube-api-access-s7rcq\" (UniqueName: \"kubernetes.io/projected/a0126455-bcc3-4334-a73c-e53b3db5bce9-kube-api-access-s7rcq\") pod \"a0126455-bcc3-4334-a73c-e53b3db5bce9\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.952886    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54d3a0c-8652-4445-b6e6-52d670571cfe-kube-api-access-c9jsq\" (UniqueName: \"kubernetes.io/projected/e54d3a0c-8652-4445-b6e6-52d670571cfe-kube-api-access-c9jsq\") pod \"e54d3a0c-8652-4445-b6e6-52d670571cfe\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.952915    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-model-cache\") pod \"e54d3a0c-8652-4445-b6e6-52d670571cfe\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.953020    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0126455-bcc3-4334-a73c-e53b3db5bce9-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0126455-bcc3-4334-a73c-e53b3db5bce9-tls-certs\") pod \"a0126455-bcc3-4334-a73c-e53b3db5bce9\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.953053    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-dshm\") pod \"e54d3a0c-8652-4445-b6e6-52d670571cfe\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.953150    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-tmp-dir\") pod \"e54d3a0c-8652-4445-b6e6-52d670571cfe\" (UID: \"e54d3a0c-8652-4445-b6e6-52d670571cfe\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.953201    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-dshm\") pod \"a0126455-bcc3-4334-a73c-e53b3db5bce9\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.953318    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-tmp-dir\") pod \"a0126455-bcc3-4334-a73c-e53b3db5bce9\" (UID: \"a0126455-bcc3-4334-a73c-e53b3db5bce9\") "
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.956323    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-tmp-dir" pod "a0126455-bcc3-4334-a73c-e53b3db5bce9" (UID: "a0126455-bcc3-4334-a73c-e53b3db5bce9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.956868    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-model-cache" pod "e54d3a0c-8652-4445-b6e6-52d670571cfe" (UID: "e54d3a0c-8652-4445-b6e6-52d670571cfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.957430    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-tmp-dir" pod "e54d3a0c-8652-4445-b6e6-52d670571cfe" (UID: "e54d3a0c-8652-4445-b6e6-52d670571cfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.957969    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-model-cache" pod "a0126455-bcc3-4334-a73c-e53b3db5bce9" (UID: "a0126455-bcc3-4334-a73c-e53b3db5bce9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.958470    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-home" pod "e54d3a0c-8652-4445-b6e6-52d670571cfe" (UID: "e54d3a0c-8652-4445-b6e6-52d670571cfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:14.961507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.958659    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-home" pod "a0126455-bcc3-4334-a73c-e53b3db5bce9" (UID: "a0126455-bcc3-4334-a73c-e53b3db5bce9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:14.975356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.975308    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-dshm" pod "e54d3a0c-8652-4445-b6e6-52d670571cfe" (UID: "e54d3a0c-8652-4445-b6e6-52d670571cfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:14.975674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.975642    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54d3a0c-8652-4445-b6e6-52d670571cfe-tls-certs" pod "e54d3a0c-8652-4445-b6e6-52d670571cfe" (UID: "e54d3a0c-8652-4445-b6e6-52d670571cfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:14.976115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.976081    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54d3a0c-8652-4445-b6e6-52d670571cfe-kube-api-access-c9jsq" pod "e54d3a0c-8652-4445-b6e6-52d670571cfe" (UID: "e54d3a0c-8652-4445-b6e6-52d670571cfe"). InnerVolumeSpecName "kube-api-access-c9jsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:14.977247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.977211    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0126455-bcc3-4334-a73c-e53b3db5bce9-kube-api-access-s7rcq" pod "a0126455-bcc3-4334-a73c-e53b3db5bce9" (UID: "a0126455-bcc3-4334-a73c-e53b3db5bce9"). InnerVolumeSpecName "kube-api-access-s7rcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:14.978673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.978647    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0126455-bcc3-4334-a73c-e53b3db5bce9-tls-certs" pod "a0126455-bcc3-4334-a73c-e53b3db5bce9" (UID: "a0126455-bcc3-4334-a73c-e53b3db5bce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:14.979377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:14.979355    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-dshm" pod "a0126455-bcc3-4334-a73c-e53b3db5bce9" (UID: "a0126455-bcc3-4334-a73c-e53b3db5bce9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:15.055250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055215    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055248    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rcq\" (UniqueName: \"kubernetes.io/projected/a0126455-bcc3-4334-a73c-e53b3db5bce9-kube-api-access-s7rcq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055261    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jsq\" (UniqueName: \"kubernetes.io/projected/e54d3a0c-8652-4445-b6e6-52d670571cfe-kube-api-access-c9jsq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055272    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055281    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0126455-bcc3-4334-a73c-e53b3db5bce9-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055326    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055334    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055342    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055348    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055355    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54d3a0c-8652-4445-b6e6-52d670571cfe-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055363    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54d3a0c-8652-4445-b6e6-52d670571cfe-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.055537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.055372    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0126455-bcc3-4334-a73c-e53b3db5bce9-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:15.118631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.118545    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9" podUID="f66f9018-58ee-4d09-9ed7-0e20a4b6700c" containerName="payload-processing" containerID="cri-o://5970461acdf68562a4ca5f189f90dbe0d10daae4314e661479e4dbc81b59380c" gracePeriod=25
Nov 02 01:49:15.910952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.910918    4851 generic.go:356] "Generic (PLEG): container finished" podID="4c69e684-14de-4c9a-9166-3e4cacc890bc" containerID="d146bc6e9ddb9e9ef8a3faf62eb0068e35ee87fd9bccbb709112ca5b4ef89c03" exitCode=0
Nov 02 01:49:15.911450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.911005    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc" event={"ID":"4c69e684-14de-4c9a-9166-3e4cacc890bc","Type":"ContainerDied","Data":"d146bc6e9ddb9e9ef8a3faf62eb0068e35ee87fd9bccbb709112ca5b4ef89c03"}
Nov 02 01:49:15.912454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.912436    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"
Nov 02 01:49:15.912575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.912463    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv" event={"ID":"a0126455-bcc3-4334-a73c-e53b3db5bce9","Type":"ContainerDied","Data":"4d2188b1844f80635ef65eeabd7ec7fbe847818b81f4a24be000c7b9cb478fb6"}
Nov 02 01:49:15.912575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.912504    4851 scope.go:122] "RemoveContainer" containerID="cc4a2c0c9fe2d30255ea7bd57d6186a63b55c9ddd70f9c6d8de895f7a3e85510"
Nov 02 01:49:15.913986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.913961    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn" event={"ID":"e54d3a0c-8652-4445-b6e6-52d670571cfe","Type":"ContainerDied","Data":"44c676b5234e3c10450d74449c0dac57d52ffcdfddc7a9e95758a6064400368e"}
Nov 02 01:49:15.913986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.913978    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"
Nov 02 01:49:15.924625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.924608    4851 scope.go:122] "RemoveContainer" containerID="85feb868483c9a8f77a74a7c6028b558a75ffeb098f7033fba35983ce5e7ca7a"
Nov 02 01:49:15.946150 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.946125    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"]
Nov 02 01:49:15.949377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.949346    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/simulator-w3-5ed42b-kserve-656db64dbd-qn5fv"]
Nov 02 01:49:15.961053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.961034    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"]
Nov 02 01:49:15.964541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:15.964521    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5ed42b/distinct-2-w3-5ed42b-kserve-785655c547-tdnsn"]
Nov 02 01:49:16.861781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:16.861741    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" path="/var/lib/kubelet/pods/a0126455-bcc3-4334-a73c-e53b3db5bce9/volumes"
Nov 02 01:49:16.862417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:16.862394    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" path="/var/lib/kubelet/pods/e54d3a0c-8652-4445-b6e6-52d670571cfe/volumes"
Nov 02 01:49:17.076388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.076366    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:17.180996 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.180964    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-xks-ca\") pod \"4c69e684-14de-4c9a-9166-3e4cacc890bc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") "
Nov 02 01:49:17.181181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.181003    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c69e684-14de-4c9a-9166-3e4cacc890bc-kube-api-access-jts6l\" (UniqueName: \"kubernetes.io/projected/4c69e684-14de-4c9a-9166-3e4cacc890bc-kube-api-access-jts6l\") pod \"4c69e684-14de-4c9a-9166-3e4cacc890bc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") "
Nov 02 01:49:17.181181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.181089    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-openshift-ca\") pod \"4c69e684-14de-4c9a-9166-3e4cacc890bc\" (UID: \"4c69e684-14de-4c9a-9166-3e4cacc890bc\") "
Nov 02 01:49:17.181531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.181508    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-openshift-ca" pod "4c69e684-14de-4c9a-9166-3e4cacc890bc" (UID: "4c69e684-14de-4c9a-9166-3e4cacc890bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:17.183328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.183264    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-xks-ca" pod "4c69e684-14de-4c9a-9166-3e4cacc890bc" (UID: "4c69e684-14de-4c9a-9166-3e4cacc890bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:17.183427 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.183367    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c69e684-14de-4c9a-9166-3e4cacc890bc-kube-api-access-jts6l" pod "4c69e684-14de-4c9a-9166-3e4cacc890bc" (UID: "4c69e684-14de-4c9a-9166-3e4cacc890bc"). InnerVolumeSpecName "kube-api-access-jts6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:17.282670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.282637    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.282670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.282668    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c69e684-14de-4c9a-9166-3e4cacc890bc-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.282874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.282681    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jts6l\" (UniqueName: \"kubernetes.io/projected/4c69e684-14de-4c9a-9166-3e4cacc890bc-kube-api-access-jts6l\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.509033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.508996    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9" podUID="f66f9018-58ee-4d09-9ed7-0e20a4b6700c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:17.624697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.624672    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:49:17.660628 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:17.660591    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:49:17.687081 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.687041    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istiod-ca-cert\") pod \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") "
Nov 02 01:49:17.687330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.687102    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-credential-socket\") pod \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") "
Nov 02 01:49:17.687330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.687135    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-data\") pod \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") "
Nov 02 01:49:17.687330 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.687175    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-kube-api-access-ckrs7\" (UniqueName: \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-kube-api-access-ckrs7\") pod \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") "
Nov 02 01:49:17.687527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.687329    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-certs\") pod \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") "
Nov 02 01:49:17.687527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.687379    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-envoy\") pod \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") "
Nov 02 01:49:17.687527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.687404    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-credential-socket" pod "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" (UID: "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:17.687527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.687413    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-podinfo\") pod \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") "
Nov 02 01:49:17.687527 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.687509    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-token\" (UniqueName: \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-token\") pod \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") "
Nov 02 01:49:17.687817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.687566    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-socket\") pod \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\" (UID: \"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5\") "
Nov 02 01:49:17.688313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.688026    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.688313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.688097    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-certs" pod "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" (UID: "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:17.688313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.688221    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-data" pod "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" (UID: "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:17.688313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.688235    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-socket" pod "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" (UID: "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:17.688626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.688564    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istiod-ca-cert" pod "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" (UID: "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:17.690065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.690030    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-podinfo" pod "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" (UID: "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:49:17.690278 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.690261    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-token" pod "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" (UID: "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:17.690479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.690462    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-kube-api-access-ckrs7" pod "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" (UID: "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5"). InnerVolumeSpecName "kube-api-access-ckrs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:17.690653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.690627    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-envoy" pod "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" (UID: "fa52a313-2e1d-4ee8-b80b-f89d0db88ff5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:17.789426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.789341    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.789426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.789370    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.789426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.789380    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.789426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.789389    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.789426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.789398    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.789426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.789408    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.789426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.789416    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.789426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.789424    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckrs7\" (UniqueName: \"kubernetes.io/projected/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5-kube-api-access-ckrs7\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:17.932160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.932111    4851 generic.go:356] "Generic (PLEG): container finished" podID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerID="dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad" exitCode=0
Nov 02 01:49:17.932399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.932174    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"
Nov 02 01:49:17.932399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.932190    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" event={"ID":"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5","Type":"ContainerDied","Data":"dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad"}
Nov 02 01:49:17.932399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.932260    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82" event={"ID":"fa52a313-2e1d-4ee8-b80b-f89d0db88ff5","Type":"ContainerDied","Data":"db2e6e38cc15b1f3fef1d40e07fa29dc12b599bf1fb74d9fdf64b772ce52c870"}
Nov 02 01:49:17.932399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.932320    4851 scope.go:122] "RemoveContainer" containerID="dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad"
Nov 02 01:49:17.935525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.935501    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc" event={"ID":"4c69e684-14de-4c9a-9166-3e4cacc890bc","Type":"ContainerDied","Data":"e8e212eea30babfcd9378695687fbae89bcfcaaa2af909dc55035effa5fb912c"}
Nov 02 01:49:17.935525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.935516    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"
Nov 02 01:49:17.935683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.935549    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8e212eea30babfcd9378695687fbae89bcfcaaa2af909dc55035effa5fb912c"
Nov 02 01:49:17.947329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.945450    4851 scope.go:122] "RemoveContainer" containerID="dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad"
Nov 02 01:49:17.947329 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:17.945907    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad\": container with ID starting with dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad not found: ID does not exist" containerID="dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad"
Nov 02 01:49:17.947329 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.945941    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad"} err="failed to get container status \"dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad\": rpc error: code = NotFound desc = could not find container \"dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad\": container with ID starting with dde02bcc38be9504efee584a4aa072b348ad2368dc7ae2173736d79343924aad not found: ID does not exist"
Nov 02 01:49:17.947971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.947952    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"]
Nov 02 01:49:17.962581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.962556    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"]
Nov 02 01:49:17.969338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.969311    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"]
Nov 02 01:49:17.976261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.976233    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-vkl82"]
Nov 02 01:49:17.978810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:17.978785    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"]
Nov 02 01:49:18.845105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:18.845075    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" path="/var/lib/kubelet/pods/fa52a313-2e1d-4ee8-b80b-f89d0db88ff5/volumes"
Nov 02 01:49:19.696529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:19.696489    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw" podUID="1c6fddd1-829f-4aef-88d8-86ecab8099c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:20.858273 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:20.858236    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"]
Nov 02 01:49:20.858776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:20.858493    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy" containerID="cri-o://d9804f6c59883965b505eb574f0f0aa4a4bb8974b4c4637856811ad53677f080" gracePeriod=30
Nov 02 01:49:21.601175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.601135    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"]
Nov 02 01:49:21.606057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.606029    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5ed42b-w9gvt"]
Nov 02 01:49:21.730662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.730618    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"]
Nov 02 01:49:21.731313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731257    4851 cpu_manager.go:414] "containerMap: removing container" podUID="77229192-c71d-4a53-becc-3338c7de9c81" containerName="payload-pre-processing"
Nov 02 01:49:21.731313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731280    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77229192-c71d-4a53-becc-3338c7de9c81" containerName="payload-pre-processing"
Nov 02 01:49:21.731313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731313    4851 cpu_manager.go:414] "containerMap: removing container" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerName="istio-proxy"
Nov 02 01:49:21.731313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731319    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerName="istio-proxy"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731328    4851 cpu_manager.go:414] "containerMap: removing container" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731333    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731341    4851 cpu_manager.go:414] "containerMap: removing container" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731345    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731356    4851 cpu_manager.go:414] "containerMap: removing container" podUID="4c69e684-14de-4c9a-9166-3e4cacc890bc" containerName="revoke-keys"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731363    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c69e684-14de-4c9a-9166-3e4cacc890bc" containerName="revoke-keys"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731373    4851 cpu_manager.go:414] "containerMap: removing container" podUID="31addff9-dc30-4d8a-8753-8229f834003f" containerName="payload-processing"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731378    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31addff9-dc30-4d8a-8753-8229f834003f" containerName="payload-processing"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731488    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="77229192-c71d-4a53-becc-3338c7de9c81" containerName="payload-pre-processing"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731495    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa52a313-2e1d-4ee8-b80b-f89d0db88ff5" containerName="istio-proxy"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731502    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54d3a0c-8652-4445-b6e6-52d670571cfe" containerName="main"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731511    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0126455-bcc3-4334-a73c-e53b3db5bce9" containerName="main"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731519    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c69e684-14de-4c9a-9166-3e4cacc890bc" containerName="revoke-keys"
Nov 02 01:49:21.731577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.731527    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="31addff9-dc30-4d8a-8753-8229f834003f" containerName="payload-processing"
Nov 02 01:49:21.736302 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.736273    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:21.740242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.740203    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:49:21.740380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.740240    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:49:21.742454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.742435    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"]
Nov 02 01:49:21.761747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.761718    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:49:21.761901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.761790    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:21.845152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.845125    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:21.845277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.845157    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nv4h\" (UniqueName: \"kubernetes.io/projected/053d5001-a748-4c58-a682-dafb91cacbb3-kube-api-access-4nv4h\") pod \"maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:21.845277 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.845188    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:21.946148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.946110    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:21.946148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.946150    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nv4h\" (UniqueName: \"kubernetes.io/projected/053d5001-a748-4c58-a682-dafb91cacbb3-kube-api-access-4nv4h\") pod \"maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:21.946683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.946173    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:21.946929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.946909    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:21.949250 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.948740    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:21.959011 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:21.958982    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nv4h\" (UniqueName: \"kubernetes.io/projected/053d5001-a748-4c58-a682-dafb91cacbb3-kube-api-access-4nv4h\") pod \"maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:22.054076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:22.054046    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:22.367653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:22.367627    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"]
Nov 02 01:49:22.369910 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:49:22.369879    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053d5001_a748_4c58_a682_dafb91cacbb3.slice/crio-28f76e9f702beb9d9132577cab849cd6f926f447f01349a581be53756df156f4 WatchSource:0}: Error finding container 28f76e9f702beb9d9132577cab849cd6f926f447f01349a581be53756df156f4: Status 404 returned error can't find the container with id 28f76e9f702beb9d9132577cab849cd6f926f447f01349a581be53756df156f4
Nov 02 01:49:22.845546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:22.845507    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f028fa5-9865-46ff-b18e-7e738d8c3ef1" path="/var/lib/kubelet/pods/9f028fa5-9865-46ff-b18e-7e738d8c3ef1/volumes"
Nov 02 01:49:23.011511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:23.011473    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr" event={"ID":"053d5001-a748-4c58-a682-dafb91cacbb3","Type":"ContainerStarted","Data":"6b3268947707c3cdc83983e16419fa0de332807bbcb43510e26a3d2f8e4b688b"}
Nov 02 01:49:23.011930 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:23.011538    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr" event={"ID":"053d5001-a748-4c58-a682-dafb91cacbb3","Type":"ContainerStarted","Data":"28f76e9f702beb9d9132577cab849cd6f926f447f01349a581be53756df156f4"}
Nov 02 01:49:23.030731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:23.030683    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr" podStartSLOduration=2.030671248 podStartE2EDuration="2.030671248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:23.029345967 +0000 UTC m=+2068.334278796" watchObservedRunningTime="2026-11-02 01:49:23.030671248 +0000 UTC m=+2068.335604073"
Nov 02 01:49:23.609568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:23.609529    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"]
Nov 02 01:49:23.624545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:23.624515    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"]
Nov 02 01:49:23.992304 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:23.992256    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"]
Nov 02 01:49:23.996965 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:23.996938    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.005453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.005425    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"]
Nov 02 01:49:24.022209 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.022180    4851 generic.go:356] "Generic (PLEG): container finished" podID="053d5001-a748-4c58-a682-dafb91cacbb3" containerID="6b3268947707c3cdc83983e16419fa0de332807bbcb43510e26a3d2f8e4b688b" exitCode=0
Nov 02 01:49:24.022602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.022219    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr" event={"ID":"053d5001-a748-4c58-a682-dafb91cacbb3","Type":"ContainerDied","Data":"6b3268947707c3cdc83983e16419fa0de332807bbcb43510e26a3d2f8e4b688b"}
Nov 02 01:49:24.086603 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.086573    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jff\" (UniqueName: \"kubernetes.io/projected/94b403e6-e800-4423-b28b-4a4c2a7831ae-kube-api-access-n6jff\") pod \"maas-api-revoke-sub-5d4722a5-95dzf\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.086785 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.086700    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d4722a5-95dzf\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.086844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.086803    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d4722a5-95dzf\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.187520 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.187487    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d4722a5-95dzf\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.187714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.187620    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d4722a5-95dzf\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.187714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.187675    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jff\" (UniqueName: \"kubernetes.io/projected/94b403e6-e800-4423-b28b-4a4c2a7831ae-kube-api-access-n6jff\") pod \"maas-api-revoke-sub-5d4722a5-95dzf\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.188366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.188345    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d4722a5-95dzf\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.190211 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.190188    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d4722a5-95dzf\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.200013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.199993    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jff\" (UniqueName: \"kubernetes.io/projected/94b403e6-e800-4423-b28b-4a4c2a7831ae-kube-api-access-n6jff\") pod \"maas-api-revoke-sub-5d4722a5-95dzf\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.315198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.315115    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:24.621844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:24.621818    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"]
Nov 02 01:49:24.623894 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:49:24.623866    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b403e6_e800_4423_b28b_4a4c2a7831ae.slice/crio-056a6721694c3ccdfda045728e1ada9811bfb71d9b5e6051ef4d1f0f7174b06e WatchSource:0}: Error finding container 056a6721694c3ccdfda045728e1ada9811bfb71d9b5e6051ef4d1f0f7174b06e: Status 404 returned error can't find the container with id 056a6721694c3ccdfda045728e1ada9811bfb71d9b5e6051ef4d1f0f7174b06e
Nov 02 01:49:25.030737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.030691    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf" event={"ID":"94b403e6-e800-4423-b28b-4a4c2a7831ae","Type":"ContainerStarted","Data":"8014d174186ca9ac8e7399873f69922a4f19c0289f74ef6e1d80f097b6cb92ef"}
Nov 02 01:49:25.031156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.030764    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf" event={"ID":"94b403e6-e800-4423-b28b-4a4c2a7831ae","Type":"ContainerStarted","Data":"056a6721694c3ccdfda045728e1ada9811bfb71d9b5e6051ef4d1f0f7174b06e"}
Nov 02 01:49:25.075327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.075160    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf" podStartSLOduration=2.075135749 podStartE2EDuration="2.075135749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:25.054911064 +0000 UTC m=+2070.359843893" watchObservedRunningTime="2026-11-02 01:49:25.075135749 +0000 UTC m=+2070.380068578"
Nov 02 01:49:25.184667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.184638    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:25.300882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.300848    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053d5001-a748-4c58-a682-dafb91cacbb3-kube-api-access-4nv4h\" (UniqueName: \"kubernetes.io/projected/053d5001-a748-4c58-a682-dafb91cacbb3-kube-api-access-4nv4h\") pod \"053d5001-a748-4c58-a682-dafb91cacbb3\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") "
Nov 02 01:49:25.301041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.300961    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-openshift-ca\") pod \"053d5001-a748-4c58-a682-dafb91cacbb3\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") "
Nov 02 01:49:25.301041 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.301004    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-xks-ca\") pod \"053d5001-a748-4c58-a682-dafb91cacbb3\" (UID: \"053d5001-a748-4c58-a682-dafb91cacbb3\") "
Nov 02 01:49:25.301414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.301380    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-openshift-ca" pod "053d5001-a748-4c58-a682-dafb91cacbb3" (UID: "053d5001-a748-4c58-a682-dafb91cacbb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:25.301529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.301510    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:25.303222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.303198    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053d5001-a748-4c58-a682-dafb91cacbb3-kube-api-access-4nv4h" pod "053d5001-a748-4c58-a682-dafb91cacbb3" (UID: "053d5001-a748-4c58-a682-dafb91cacbb3"). InnerVolumeSpecName "kube-api-access-4nv4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:25.303357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.303267    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-xks-ca" pod "053d5001-a748-4c58-a682-dafb91cacbb3" (UID: "053d5001-a748-4c58-a682-dafb91cacbb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:25.402175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.402138    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nv4h\" (UniqueName: \"kubernetes.io/projected/053d5001-a748-4c58-a682-dafb91cacbb3-kube-api-access-4nv4h\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:25.402175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:25.402171    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053d5001-a748-4c58-a682-dafb91cacbb3-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:26.073639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.068501    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr" event={"ID":"053d5001-a748-4c58-a682-dafb91cacbb3","Type":"ContainerDied","Data":"28f76e9f702beb9d9132577cab849cd6f926f447f01349a581be53756df156f4"}
Nov 02 01:49:26.073639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.068565    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28f76e9f702beb9d9132577cab849cd6f926f447f01349a581be53756df156f4"
Nov 02 01:49:26.073639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.068646    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"
Nov 02 01:49:26.076448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.076406    4851 generic.go:356] "Generic (PLEG): container finished" podID="94b403e6-e800-4423-b28b-4a4c2a7831ae" containerID="8014d174186ca9ac8e7399873f69922a4f19c0289f74ef6e1d80f097b6cb92ef" exitCode=0
Nov 02 01:49:26.076588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.076540    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf" event={"ID":"94b403e6-e800-4423-b28b-4a4c2a7831ae","Type":"ContainerDied","Data":"8014d174186ca9ac8e7399873f69922a4f19c0289f74ef6e1d80f097b6cb92ef"}
Nov 02 01:49:26.078902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.078879    4851 generic.go:356] "Generic (PLEG): container finished" podID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerID="d9804f6c59883965b505eb574f0f0aa4a4bb8974b4c4637856811ad53677f080" exitCode=0
Nov 02 01:49:26.079084 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.079058    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" event={"ID":"d486ac0b-a945-45aa-b47b-9ece667cb1f1","Type":"ContainerDied","Data":"d9804f6c59883965b505eb574f0f0aa4a4bb8974b4c4637856811ad53677f080"}
Nov 02 01:49:26.079175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.079129    4851 scope.go:122] "RemoveContainer" containerID="4c4f2f98196832046611760edc552add6b53e824ea6b039127289953dbcdfbe7"
Nov 02 01:49:26.215853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.215818    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:49:26.311981 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.311946    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"]
Nov 02 01:49:26.314937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.314912    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-data\") pod \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") "
Nov 02 01:49:26.315082 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.314948    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-kube-api-access-np954\" (UniqueName: \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-kube-api-access-np954\") pod \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") "
Nov 02 01:49:26.315082 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.314968    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-socket\") pod \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") "
Nov 02 01:49:26.315082 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.314992    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-credential-socket\") pod \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") "
Nov 02 01:49:26.315082 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315060    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-envoy\") pod \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") "
Nov 02 01:49:26.315397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315098    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-token\" (UniqueName: \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-token\") pod \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") "
Nov 02 01:49:26.315397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315125    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-podinfo\") pod \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") "
Nov 02 01:49:26.315397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315178    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-certs\") pod \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") "
Nov 02 01:49:26.315397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315234    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-data" pod "d486ac0b-a945-45aa-b47b-9ece667cb1f1" (UID: "d486ac0b-a945-45aa-b47b-9ece667cb1f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:26.315397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315340    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istiod-ca-cert\") pod \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\" (UID: \"d486ac0b-a945-45aa-b47b-9ece667cb1f1\") "
Nov 02 01:49:26.316208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315459    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-credential-socket" pod "d486ac0b-a945-45aa-b47b-9ece667cb1f1" (UID: "d486ac0b-a945-45aa-b47b-9ece667cb1f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:26.316208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315528    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-certs" pod "d486ac0b-a945-45aa-b47b-9ece667cb1f1" (UID: "d486ac0b-a945-45aa-b47b-9ece667cb1f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:26.316208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315601    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-socket" pod "d486ac0b-a945-45aa-b47b-9ece667cb1f1" (UID: "d486ac0b-a945-45aa-b47b-9ece667cb1f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:26.316208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315867    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istiod-ca-cert" pod "d486ac0b-a945-45aa-b47b-9ece667cb1f1" (UID: "d486ac0b-a945-45aa-b47b-9ece667cb1f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:26.316208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.315997    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:26.316208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.316019    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:26.316208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.316035    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:26.316208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.316050    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:26.317396 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.317371    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-kube-api-access-np954" pod "d486ac0b-a945-45aa-b47b-9ece667cb1f1" (UID: "d486ac0b-a945-45aa-b47b-9ece667cb1f1"). InnerVolumeSpecName "kube-api-access-np954". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:26.317507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.317483    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-podinfo" pod "d486ac0b-a945-45aa-b47b-9ece667cb1f1" (UID: "d486ac0b-a945-45aa-b47b-9ece667cb1f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:49:26.317866 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.317850    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-token" pod "d486ac0b-a945-45aa-b47b-9ece667cb1f1" (UID: "d486ac0b-a945-45aa-b47b-9ece667cb1f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:26.319048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.319028    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-envoy" pod "d486ac0b-a945-45aa-b47b-9ece667cb1f1" (UID: "d486ac0b-a945-45aa-b47b-9ece667cb1f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:26.417442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.417404    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:26.417442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.417437    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:26.417442 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.417450    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:26.417694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.417465    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np954\" (UniqueName: \"kubernetes.io/projected/d486ac0b-a945-45aa-b47b-9ece667cb1f1-kube-api-access-np954\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:26.417694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:26.417479    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d486ac0b-a945-45aa-b47b-9ece667cb1f1-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:27.086779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.086751    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"
Nov 02 01:49:27.087252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.086783    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv" event={"ID":"d486ac0b-a945-45aa-b47b-9ece667cb1f1","Type":"ContainerDied","Data":"dd99e36d16eb647473b1cade3f431f6339259c01544bccdcee0d4091c4f4e00a"}
Nov 02 01:49:27.087252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.086878    4851 scope.go:122] "RemoveContainer" containerID="d9804f6c59883965b505eb574f0f0aa4a4bb8974b4c4637856811ad53677f080"
Nov 02 01:49:27.115467 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.115366    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"]
Nov 02 01:49:27.118755 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.118711    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-2xfvv"]
Nov 02 01:49:27.247481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.247451    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:27.336913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.336881    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-xks-ca\") pod \"94b403e6-e800-4423-b28b-4a4c2a7831ae\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") "
Nov 02 01:49:27.337138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.336932    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b403e6-e800-4423-b28b-4a4c2a7831ae-kube-api-access-n6jff\" (UniqueName: \"kubernetes.io/projected/94b403e6-e800-4423-b28b-4a4c2a7831ae-kube-api-access-n6jff\") pod \"94b403e6-e800-4423-b28b-4a4c2a7831ae\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") "
Nov 02 01:49:27.337138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.337018    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-openshift-ca\") pod \"94b403e6-e800-4423-b28b-4a4c2a7831ae\" (UID: \"94b403e6-e800-4423-b28b-4a4c2a7831ae\") "
Nov 02 01:49:27.337434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.337410    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-openshift-ca" pod "94b403e6-e800-4423-b28b-4a4c2a7831ae" (UID: "94b403e6-e800-4423-b28b-4a4c2a7831ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:27.339685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.339657    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-xks-ca" pod "94b403e6-e800-4423-b28b-4a4c2a7831ae" (UID: "94b403e6-e800-4423-b28b-4a4c2a7831ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:27.339803 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.339701    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b403e6-e800-4423-b28b-4a4c2a7831ae-kube-api-access-n6jff" pod "94b403e6-e800-4423-b28b-4a4c2a7831ae" (UID: "94b403e6-e800-4423-b28b-4a4c2a7831ae"). InnerVolumeSpecName "kube-api-access-n6jff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:27.438529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.438488    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:27.438529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.438525    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jff\" (UniqueName: \"kubernetes.io/projected/94b403e6-e800-4423-b28b-4a4c2a7831ae-kube-api-access-n6jff\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:27.438529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.438537    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b403e6-e800-4423-b28b-4a4c2a7831ae-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:27.509276 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:27.509237    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9" podUID="f66f9018-58ee-4d09-9ed7-0e20a4b6700c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:28.021584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:28.021548    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"]
Nov 02 01:49:28.024139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:28.024116    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d167dadf-5rlrg"]
Nov 02 01:49:28.095104 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:28.095076    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"
Nov 02 01:49:28.095544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:28.095073    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf" event={"ID":"94b403e6-e800-4423-b28b-4a4c2a7831ae","Type":"ContainerDied","Data":"056a6721694c3ccdfda045728e1ada9811bfb71d9b5e6051ef4d1f0f7174b06e"}
Nov 02 01:49:28.095544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:28.095202    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="056a6721694c3ccdfda045728e1ada9811bfb71d9b5e6051ef4d1f0f7174b06e"
Nov 02 01:49:28.609967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:28.609925    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln" podUID="c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" containerName="payload-processing" containerID="cri-o://281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b" gracePeriod=25
Nov 02 01:49:28.625089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:28.625044    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8" podUID="c6ac98d6-c879-4606-81a4-b791e4af5b3f" containerName="payload-pre-processing" containerID="cri-o://54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d" gracePeriod=25
Nov 02 01:49:28.632776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:28.632745    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8" podUID="c6ac98d6-c879-4606-81a4-b791e4af5b3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:28.846952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:28.846917    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bd2122-f60f-47c4-b6ce-13a555092662" path="/var/lib/kubelet/pods/a9bd2122-f60f-47c4-b6ce-13a555092662/volumes"
Nov 02 01:49:28.847497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:28.847475    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" path="/var/lib/kubelet/pods/d486ac0b-a945-45aa-b47b-9ece667cb1f1/volumes"
Nov 02 01:49:29.012537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:29.011766    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"]
Nov 02 01:49:29.012537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:29.012052    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy" containerID="cri-o://a8f8284f26f5e5cf2a3a6c68a558b360b4cc5baff4d458bbe6edcf1842927494" gracePeriod=30
Nov 02 01:49:29.695915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:29.695870    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln" podUID="c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:29.696332 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:29.696276    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw" podUID="1c6fddd1-829f-4aef-88d8-86ecab8099c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:30.444149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:30.444089    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:49:30.591248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:30.591217    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6fddd1-829f-4aef-88d8-86ecab8099c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6fddd1-829f-4aef-88d8-86ecab8099c6-plugins-config-volume\") pod \"1c6fddd1-829f-4aef-88d8-86ecab8099c6\" (UID: \"1c6fddd1-829f-4aef-88d8-86ecab8099c6\") "
Nov 02 01:49:30.591438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:30.591368    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6fddd1-829f-4aef-88d8-86ecab8099c6-kube-api-access-wjmv5\" (UniqueName: \"kubernetes.io/projected/1c6fddd1-829f-4aef-88d8-86ecab8099c6-kube-api-access-wjmv5\") pod \"1c6fddd1-829f-4aef-88d8-86ecab8099c6\" (UID: \"1c6fddd1-829f-4aef-88d8-86ecab8099c6\") "
Nov 02 01:49:30.591619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:30.591596    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6fddd1-829f-4aef-88d8-86ecab8099c6-plugins-config-volume" pod "1c6fddd1-829f-4aef-88d8-86ecab8099c6" (UID: "1c6fddd1-829f-4aef-88d8-86ecab8099c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:30.591720 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:30.591708    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c6fddd1-829f-4aef-88d8-86ecab8099c6-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:30.593702 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:30.593674    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6fddd1-829f-4aef-88d8-86ecab8099c6-kube-api-access-wjmv5" pod "1c6fddd1-829f-4aef-88d8-86ecab8099c6" (UID: "1c6fddd1-829f-4aef-88d8-86ecab8099c6"). InnerVolumeSpecName "kube-api-access-wjmv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:30.692417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:30.692386    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjmv5\" (UniqueName: \"kubernetes.io/projected/1c6fddd1-829f-4aef-88d8-86ecab8099c6-kube-api-access-wjmv5\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:31.152409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.152327    4851 generic.go:356] "Generic (PLEG): container finished" podID="1c6fddd1-829f-4aef-88d8-86ecab8099c6" containerID="f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772" exitCode=0
Nov 02 01:49:31.152409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.152372    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"
Nov 02 01:49:31.152973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.152415    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw" event={"ID":"1c6fddd1-829f-4aef-88d8-86ecab8099c6","Type":"ContainerDied","Data":"f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772"}
Nov 02 01:49:31.152973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.152461    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw" event={"ID":"1c6fddd1-829f-4aef-88d8-86ecab8099c6","Type":"ContainerDied","Data":"1187d4d4ea7e54e819c2be92d861170c129a1ad56c9c5b95bff88c6ba2f62e0e"}
Nov 02 01:49:31.152973 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.152484    4851 scope.go:122] "RemoveContainer" containerID="f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772"
Nov 02 01:49:31.163619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.163580    4851 scope.go:122] "RemoveContainer" containerID="f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772"
Nov 02 01:49:31.163844 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:31.163823    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772\": container with ID starting with f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772 not found: ID does not exist" containerID="f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772"
Nov 02 01:49:31.163906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.163852    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772"} err="failed to get container status \"f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772\": rpc error: code = NotFound desc = could not find container \"f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772\": container with ID starting with f5ecadc6733b6287a55ffbf554f94374458e0d9bd8fa53b9dd8191a8ae6e6772 not found: ID does not exist"
Nov 02 01:49:31.171626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.171596    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"]
Nov 02 01:49:31.175248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.175226    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5ed42b-7bf69f57d5-vh9vw"]
Nov 02 01:49:31.855182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.855152    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"]
Nov 02 01:49:31.909789 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.909757    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"]
Nov 02 01:49:31.910212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910200    4851 cpu_manager.go:414] "containerMap: removing container" podUID="1c6fddd1-829f-4aef-88d8-86ecab8099c6" containerName="payload-processing"
Nov 02 01:49:31.910257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910213    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6fddd1-829f-4aef-88d8-86ecab8099c6" containerName="payload-processing"
Nov 02 01:49:31.910257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910225    4851 cpu_manager.go:414] "containerMap: removing container" podUID="94b403e6-e800-4423-b28b-4a4c2a7831ae" containerName="revoke-keys"
Nov 02 01:49:31.910257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910230    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b403e6-e800-4423-b28b-4a4c2a7831ae" containerName="revoke-keys"
Nov 02 01:49:31.910257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910238    4851 cpu_manager.go:414] "containerMap: removing container" podUID="053d5001-a748-4c58-a682-dafb91cacbb3" containerName="revoke-keys"
Nov 02 01:49:31.910257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910243    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053d5001-a748-4c58-a682-dafb91cacbb3" containerName="revoke-keys"
Nov 02 01:49:31.910257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910250    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy"
Nov 02 01:49:31.910257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910254    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy"
Nov 02 01:49:31.910538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910345    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6fddd1-829f-4aef-88d8-86ecab8099c6" containerName="payload-processing"
Nov 02 01:49:31.910538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910354    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b403e6-e800-4423-b28b-4a4c2a7831ae" containerName="revoke-keys"
Nov 02 01:49:31.910538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910360    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="053d5001-a748-4c58-a682-dafb91cacbb3" containerName="revoke-keys"
Nov 02 01:49:31.910538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.910367    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy"
Nov 02 01:49:31.915017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.914992    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:31.921800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.921751    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:49:31.922157 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.922118    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:49:31.923370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.922688    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:49:31.923370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.922721    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:31.930047 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:31.928528    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"]
Nov 02 01:49:32.008995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.008962    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046c5160-z82lf\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:32.009210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.009186    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2zd2\" (UniqueName: \"kubernetes.io/projected/4cf5f50e-a1f8-43ac-9e17-163776479428-kube-api-access-j2zd2\") pod \"maas-api-revoke-sub-046c5160-z82lf\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:32.009344 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.009248    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046c5160-z82lf\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:32.110999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.110896    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2zd2\" (UniqueName: \"kubernetes.io/projected/4cf5f50e-a1f8-43ac-9e17-163776479428-kube-api-access-j2zd2\") pod \"maas-api-revoke-sub-046c5160-z82lf\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:32.110999 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.110954    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046c5160-z82lf\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:32.111499 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.111460    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046c5160-z82lf\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:32.112501 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.112476    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046c5160-z82lf\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:32.116213 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.116188    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046c5160-z82lf\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:32.142355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.142280    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2zd2\" (UniqueName: \"kubernetes.io/projected/4cf5f50e-a1f8-43ac-9e17-163776479428-kube-api-access-j2zd2\") pod \"maas-api-revoke-sub-046c5160-z82lf\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:32.234614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.234579    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:32.516020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.515981    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"]
Nov 02 01:49:32.516819 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:49:32.516789    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf5f50e_a1f8_43ac_9e17_163776479428.slice/crio-936e89f2283485f2a84d66d8c88df14963dd6ca655025ad25eac8c0fece30ef7 WatchSource:0}: Error finding container 936e89f2283485f2a84d66d8c88df14963dd6ca655025ad25eac8c0fece30ef7: Status 404 returned error can't find the container with id 936e89f2283485f2a84d66d8c88df14963dd6ca655025ad25eac8c0fece30ef7
Nov 02 01:49:32.846794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.846762    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6fddd1-829f-4aef-88d8-86ecab8099c6" path="/var/lib/kubelet/pods/1c6fddd1-829f-4aef-88d8-86ecab8099c6/volumes"
Nov 02 01:49:32.973538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.973503    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" containerID="cri-o://945de5f2a496a130b017e9a25ef8d6fe584256d53162bc0c6c9c18c5817faf15" gracePeriod=45
Nov 02 01:49:32.987699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.987671    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main" containerID="cri-o://fbb13aff88a39fc3da0c8b159d30baefcfe011caba85f9f8bdbd93f57fef556d" gracePeriod=45
Nov 02 01:49:32.999868 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:32.999840    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" containerID="cri-o://7a9b3efdd6bafb9fbbe8ddeea06ee828d5562e477105fdcb057d4806f2fd5fee" gracePeriod=45
Nov 02 01:49:33.162796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.162757    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": dial tcp 10.132.0.55:8000: connect: connection refused"
Nov 02 01:49:33.192123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.192082    4851 generic.go:356] "Generic (PLEG): container finished" podID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerID="fbb13aff88a39fc3da0c8b159d30baefcfe011caba85f9f8bdbd93f57fef556d" exitCode=0
Nov 02 01:49:33.192314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.192214    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" event={"ID":"17113a57-21c8-46c1-a80a-deb91dc51a87","Type":"ContainerDied","Data":"fbb13aff88a39fc3da0c8b159d30baefcfe011caba85f9f8bdbd93f57fef556d"}
Nov 02 01:49:33.194255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.194209    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf" event={"ID":"4cf5f50e-a1f8-43ac-9e17-163776479428","Type":"ContainerStarted","Data":"2a1589b9b7be2f160465b7291a48c7481b107142189e62a0170d0b6fc15ee1a3"}
Nov 02 01:49:33.194367 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.194261    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf" event={"ID":"4cf5f50e-a1f8-43ac-9e17-163776479428","Type":"ContainerStarted","Data":"936e89f2283485f2a84d66d8c88df14963dd6ca655025ad25eac8c0fece30ef7"}
Nov 02 01:49:33.195926 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.195905    4851 generic.go:356] "Generic (PLEG): container finished" podID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerID="945de5f2a496a130b017e9a25ef8d6fe584256d53162bc0c6c9c18c5817faf15" exitCode=0
Nov 02 01:49:33.196059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.196043    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" event={"ID":"00fd0bde-f5d1-4a50-ad48-85bc6881cb27","Type":"ContainerDied","Data":"945de5f2a496a130b017e9a25ef8d6fe584256d53162bc0c6c9c18c5817faf15"}
Nov 02 01:49:33.197575 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.197555    4851 generic.go:356] "Generic (PLEG): container finished" podID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerID="7a9b3efdd6bafb9fbbe8ddeea06ee828d5562e477105fdcb057d4806f2fd5fee" exitCode=0
Nov 02 01:49:33.197693 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.197599    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" event={"ID":"ee158de2-d2d6-46dd-b0e8-b4e1a366633e","Type":"ContainerDied","Data":"7a9b3efdd6bafb9fbbe8ddeea06ee828d5562e477105fdcb057d4806f2fd5fee"}
Nov 02 01:49:33.216771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.216725    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf" podStartSLOduration=2.21671143 podStartE2EDuration="2.21671143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:33.212384594 +0000 UTC m=+2078.517317415" watchObservedRunningTime="2026-11-02 01:49:33.21671143 +0000 UTC m=+2078.521644309"
Nov 02 01:49:33.291757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.291732    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:49:33.296331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.296308    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:49:33.334400 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.334372    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:49:33.362074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.362004    4851 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:49:33.362074 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.362059    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:49:33.426993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.426961    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-model-cache\") pod \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") "
Nov 02 01:49:33.427188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427031    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-model-cache\") pod \"17113a57-21c8-46c1-a80a-deb91dc51a87\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") "
Nov 02 01:49:33.427188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427059    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tls-certs\") pod \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") "
Nov 02 01:49:33.427188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427086    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tmp-dir\") pod \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") "
Nov 02 01:49:33.427188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427146    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17113a57-21c8-46c1-a80a-deb91dc51a87-kube-api-access-ps7vl\" (UniqueName: \"kubernetes.io/projected/17113a57-21c8-46c1-a80a-deb91dc51a87-kube-api-access-ps7vl\") pod \"17113a57-21c8-46c1-a80a-deb91dc51a87\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") "
Nov 02 01:49:33.427188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427171    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-dshm\") pod \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") "
Nov 02 01:49:33.427493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427198    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-dshm\") pod \"17113a57-21c8-46c1-a80a-deb91dc51a87\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") "
Nov 02 01:49:33.427493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427235    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-kube-api-access-dc8wh\" (UniqueName: \"kubernetes.io/projected/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-kube-api-access-dc8wh\") pod \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") "
Nov 02 01:49:33.427493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427272    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tls-certs\") pod \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") "
Nov 02 01:49:33.427493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427334    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-dshm\") pod \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") "
Nov 02 01:49:33.427493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427344    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-model-cache" pod "ee158de2-d2d6-46dd-b0e8-b4e1a366633e" (UID: "ee158de2-d2d6-46dd-b0e8-b4e1a366633e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.427493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427346    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-model-cache" pod "17113a57-21c8-46c1-a80a-deb91dc51a87" (UID: "17113a57-21c8-46c1-a80a-deb91dc51a87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.427493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427388    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-home\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-home\") pod \"17113a57-21c8-46c1-a80a-deb91dc51a87\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") "
Nov 02 01:49:33.427493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427445    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tmp-dir\") pod \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") "
Nov 02 01:49:33.427493 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427478    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tmp-dir" pod "ee158de2-d2d6-46dd-b0e8-b4e1a366633e" (UID: "ee158de2-d2d6-46dd-b0e8-b4e1a366633e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.427951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427547    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-kube-api-access-fz2zq\" (UniqueName: \"kubernetes.io/projected/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-kube-api-access-fz2zq\") pod \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") "
Nov 02 01:49:33.427951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427577    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-home\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-home\") pod \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") "
Nov 02 01:49:33.427951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427622    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-tmp-dir\") pod \"17113a57-21c8-46c1-a80a-deb91dc51a87\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") "
Nov 02 01:49:33.427951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427665    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17113a57-21c8-46c1-a80a-deb91dc51a87-tls-certs\" (UniqueName: \"kubernetes.io/secret/17113a57-21c8-46c1-a80a-deb91dc51a87-tls-certs\") pod \"17113a57-21c8-46c1-a80a-deb91dc51a87\" (UID: \"17113a57-21c8-46c1-a80a-deb91dc51a87\") "
Nov 02 01:49:33.427951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427694    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-model-cache\") pod \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\" (UID: \"00fd0bde-f5d1-4a50-ad48-85bc6881cb27\") "
Nov 02 01:49:33.427951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.427737    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-home\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-home\") pod \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\" (UID: \"ee158de2-d2d6-46dd-b0e8-b4e1a366633e\") "
Nov 02 01:49:33.428373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.428157    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.428373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.428175    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.428373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.428188    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.428550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.428475    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-home" pod "ee158de2-d2d6-46dd-b0e8-b4e1a366633e" (UID: "ee158de2-d2d6-46dd-b0e8-b4e1a366633e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.428698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.428674    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-home" pod "17113a57-21c8-46c1-a80a-deb91dc51a87" (UID: "17113a57-21c8-46c1-a80a-deb91dc51a87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.428914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.428886    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tmp-dir" pod "00fd0bde-f5d1-4a50-ad48-85bc6881cb27" (UID: "00fd0bde-f5d1-4a50-ad48-85bc6881cb27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.430395 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.430364    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tls-certs" pod "00fd0bde-f5d1-4a50-ad48-85bc6881cb27" (UID: "00fd0bde-f5d1-4a50-ad48-85bc6881cb27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:33.430517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.430416    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-dshm" pod "17113a57-21c8-46c1-a80a-deb91dc51a87" (UID: "17113a57-21c8-46c1-a80a-deb91dc51a87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.430588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.430514    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-dshm" pod "ee158de2-d2d6-46dd-b0e8-b4e1a366633e" (UID: "ee158de2-d2d6-46dd-b0e8-b4e1a366633e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.430649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.430632    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-tmp-dir" pod "17113a57-21c8-46c1-a80a-deb91dc51a87" (UID: "17113a57-21c8-46c1-a80a-deb91dc51a87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.430867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.430842    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-model-cache" pod "00fd0bde-f5d1-4a50-ad48-85bc6881cb27" (UID: "00fd0bde-f5d1-4a50-ad48-85bc6881cb27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.430867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.430855    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-home" pod "00fd0bde-f5d1-4a50-ad48-85bc6881cb27" (UID: "00fd0bde-f5d1-4a50-ad48-85bc6881cb27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.431024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.431005    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tls-certs" pod "ee158de2-d2d6-46dd-b0e8-b4e1a366633e" (UID: "ee158de2-d2d6-46dd-b0e8-b4e1a366633e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:33.431415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.431389    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17113a57-21c8-46c1-a80a-deb91dc51a87-kube-api-access-ps7vl" pod "17113a57-21c8-46c1-a80a-deb91dc51a87" (UID: "17113a57-21c8-46c1-a80a-deb91dc51a87"). InnerVolumeSpecName "kube-api-access-ps7vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:33.431523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.431417    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-kube-api-access-fz2zq" pod "ee158de2-d2d6-46dd-b0e8-b4e1a366633e" (UID: "ee158de2-d2d6-46dd-b0e8-b4e1a366633e"). InnerVolumeSpecName "kube-api-access-fz2zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:33.431523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.431479    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-kube-api-access-dc8wh" pod "00fd0bde-f5d1-4a50-ad48-85bc6881cb27" (UID: "00fd0bde-f5d1-4a50-ad48-85bc6881cb27"). InnerVolumeSpecName "kube-api-access-dc8wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:33.432166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.432150    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-dshm" pod "00fd0bde-f5d1-4a50-ad48-85bc6881cb27" (UID: "00fd0bde-f5d1-4a50-ad48-85bc6881cb27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:33.432799 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.432729    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17113a57-21c8-46c1-a80a-deb91dc51a87-tls-certs" pod "17113a57-21c8-46c1-a80a-deb91dc51a87" (UID: "17113a57-21c8-46c1-a80a-deb91dc51a87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:33.528920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.528891    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc8wh\" (UniqueName: \"kubernetes.io/projected/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-kube-api-access-dc8wh\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.528920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.528920    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.528933    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.528945    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.528956    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.528968    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz2zq\" (UniqueName: \"kubernetes.io/projected/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-kube-api-access-fz2zq\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.528979    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.528991    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.529002    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17113a57-21c8-46c1-a80a-deb91dc51a87-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.529012    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.529024    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.529037    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee158de2-d2d6-46dd-b0e8-b4e1a366633e-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.529048    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps7vl\" (UniqueName: \"kubernetes.io/projected/17113a57-21c8-46c1-a80a-deb91dc51a87-kube-api-access-ps7vl\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.529059    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fd0bde-f5d1-4a50-ad48-85bc6881cb27-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:33.529139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:33.529072    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17113a57-21c8-46c1-a80a-deb91dc51a87-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.208690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.208658    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"
Nov 02 01:49:34.208690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.208680    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" event={"ID":"00fd0bde-f5d1-4a50-ad48-85bc6881cb27","Type":"ContainerDied","Data":"895523fc9b4a2bab0005b7c3e37c15a15f31a8f333609e62fee6377c9b6a4ab3"}
Nov 02 01:49:34.208931 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.208740    4851 scope.go:122] "RemoveContainer" containerID="945de5f2a496a130b017e9a25ef8d6fe584256d53162bc0c6c9c18c5817faf15"
Nov 02 01:49:34.210875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.210847    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml" event={"ID":"ee158de2-d2d6-46dd-b0e8-b4e1a366633e","Type":"ContainerDied","Data":"9d19de129c7fe907836c8230a646bfd5a6ec9eb741dc0f2cfb79a58095cddf9c"}
Nov 02 01:49:34.211013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.210878    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"
Nov 02 01:49:34.218515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.218392    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94" event={"ID":"17113a57-21c8-46c1-a80a-deb91dc51a87","Type":"ContainerDied","Data":"e410af5f5cf142c96ac01438a7c8fd1fc1cae91a0e99fe8b8294a75106847a2f"}
Nov 02 01:49:34.218515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.218474    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"
Nov 02 01:49:34.230246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.230223    4851 scope.go:122] "RemoveContainer" containerID="7a9b3efdd6bafb9fbbe8ddeea06ee828d5562e477105fdcb057d4806f2fd5fee"
Nov 02 01:49:34.236820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.236793    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" event={"ID":"bc860c6b-3659-4fce-b6ab-401e1264fc67","Type":"ContainerDied","Data":"a8f8284f26f5e5cf2a3a6c68a558b360b4cc5baff4d458bbe6edcf1842927494"}
Nov 02 01:49:34.236939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.236859    4851 generic.go:356] "Generic (PLEG): container finished" podID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerID="a8f8284f26f5e5cf2a3a6c68a558b360b4cc5baff4d458bbe6edcf1842927494" exitCode=0
Nov 02 01:49:34.244069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.244043    4851 generic.go:356] "Generic (PLEG): container finished" podID="4cf5f50e-a1f8-43ac-9e17-163776479428" containerID="2a1589b9b7be2f160465b7291a48c7481b107142189e62a0170d0b6fc15ee1a3" exitCode=0
Nov 02 01:49:34.244177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.244129    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf" event={"ID":"4cf5f50e-a1f8-43ac-9e17-163776479428","Type":"ContainerDied","Data":"2a1589b9b7be2f160465b7291a48c7481b107142189e62a0170d0b6fc15ee1a3"}
Nov 02 01:49:34.244177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.244171    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"]
Nov 02 01:49:34.248098 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.248074    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq"]
Nov 02 01:49:34.250771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.250554    4851 scope.go:122] "RemoveContainer" containerID="fbb13aff88a39fc3da0c8b159d30baefcfe011caba85f9f8bdbd93f57fef556d"
Nov 02 01:49:34.258688 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.258659    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-2-w0-ca1c53-kserve-7b84f74758-skqjq" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:34.261203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.261182    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"]
Nov 02 01:49:34.265757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.265735    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/premium-w0-ca1c53-kserve-6f7fc6cb78-pvq94"]
Nov 02 01:49:34.278451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.277952    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"]
Nov 02 01:49:34.281842 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.281818    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/simulator-w0-ca1c53-kserve-55cdfdd467-ndqml"]
Nov 02 01:49:34.283052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.283031    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:49:34.374076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.374044    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"]
Nov 02 01:49:34.374613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.374247    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k" podUID="c1a77a15-50e0-43d2-b455-459f74de3250" containerName="discovery" containerID="cri-o://6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555" gracePeriod=30
Nov 02 01:49:34.415017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.414987    4851 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nnr4k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:8080/ready\": dial tcp 10.132.0.31:8080: connect: connection refused" start-of-body=
Nov 02 01:49:34.415140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.415046    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k" podUID="c1a77a15-50e0-43d2-b455-459f74de3250" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.31:8080/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Nov 02 01:49:34.437714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.437681    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-envoy\") pod \"bc860c6b-3659-4fce-b6ab-401e1264fc67\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") "
Nov 02 01:49:34.437905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.437743    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-token\" (UniqueName: \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-token\") pod \"bc860c6b-3659-4fce-b6ab-401e1264fc67\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") "
Nov 02 01:49:34.437905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.437779    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc860c6b-3659-4fce-b6ab-401e1264fc67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc860c6b-3659-4fce-b6ab-401e1264fc67-istiod-ca-cert\") pod \"bc860c6b-3659-4fce-b6ab-401e1264fc67\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") "
Nov 02 01:49:34.437905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.437871    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-socket\") pod \"bc860c6b-3659-4fce-b6ab-401e1264fc67\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") "
Nov 02 01:49:34.438061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.437938    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-data\") pod \"bc860c6b-3659-4fce-b6ab-401e1264fc67\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") "
Nov 02 01:49:34.438061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438025    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-credential-socket\") pod \"bc860c6b-3659-4fce-b6ab-401e1264fc67\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") "
Nov 02 01:49:34.438134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438060    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-certs\") pod \"bc860c6b-3659-4fce-b6ab-401e1264fc67\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") "
Nov 02 01:49:34.438134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438092    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-kube-api-access-5cvbx\" (UniqueName: \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-kube-api-access-5cvbx\") pod \"bc860c6b-3659-4fce-b6ab-401e1264fc67\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") "
Nov 02 01:49:34.438222 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438186    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-podinfo\") pod \"bc860c6b-3659-4fce-b6ab-401e1264fc67\" (UID: \"bc860c6b-3659-4fce-b6ab-401e1264fc67\") "
Nov 02 01:49:34.438331 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438258    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc860c6b-3659-4fce-b6ab-401e1264fc67-istiod-ca-cert" pod "bc860c6b-3659-4fce-b6ab-401e1264fc67" (UID: "bc860c6b-3659-4fce-b6ab-401e1264fc67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:34.438511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438490    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-credential-socket" pod "bc860c6b-3659-4fce-b6ab-401e1264fc67" (UID: "bc860c6b-3659-4fce-b6ab-401e1264fc67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:34.438610 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438566    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-certs" pod "bc860c6b-3659-4fce-b6ab-401e1264fc67" (UID: "bc860c6b-3659-4fce-b6ab-401e1264fc67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:34.438734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438708    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-data" pod "bc860c6b-3659-4fce-b6ab-401e1264fc67" (UID: "bc860c6b-3659-4fce-b6ab-401e1264fc67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:34.438815 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438800    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.438877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438820    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.438877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438832    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.438877 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438845    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc860c6b-3659-4fce-b6ab-401e1264fc67-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.439024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.438900    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-socket" pod "bc860c6b-3659-4fce-b6ab-401e1264fc67" (UID: "bc860c6b-3659-4fce-b6ab-401e1264fc67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:34.440607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.440579    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-token" pod "bc860c6b-3659-4fce-b6ab-401e1264fc67" (UID: "bc860c6b-3659-4fce-b6ab-401e1264fc67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:34.444741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.444709    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-kube-api-access-5cvbx" pod "bc860c6b-3659-4fce-b6ab-401e1264fc67" (UID: "bc860c6b-3659-4fce-b6ab-401e1264fc67"). InnerVolumeSpecName "kube-api-access-5cvbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:34.444840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.444799    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-podinfo" pod "bc860c6b-3659-4fce-b6ab-401e1264fc67" (UID: "bc860c6b-3659-4fce-b6ab-401e1264fc67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:49:34.444840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.444817    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-envoy" pod "bc860c6b-3659-4fce-b6ab-401e1264fc67" (UID: "bc860c6b-3659-4fce-b6ab-401e1264fc67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:34.539465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.539368    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.539465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.539405    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvbx\" (UniqueName: \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-kube-api-access-5cvbx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.539465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.539421    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.539465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.539432    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.539465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.539444    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc860c6b-3659-4fce-b6ab-401e1264fc67-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.661956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.661927    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:49:34.846946 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.846857    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-istio-token\" (UniqueName: \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-istio-token\") pod \"c1a77a15-50e0-43d2-b455-459f74de3250\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") "
Nov 02 01:49:34.847128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.846955    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-cacerts\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-cacerts\") pod \"c1a77a15-50e0-43d2-b455-459f74de3250\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") "
Nov 02 01:49:34.847128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.847051    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-dns-cert\") pod \"c1a77a15-50e0-43d2-b455-459f74de3250\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") "
Nov 02 01:49:34.847128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.847087    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-kubeconfig\") pod \"c1a77a15-50e0-43d2-b455-459f74de3250\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") "
Nov 02 01:49:34.847128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.847118    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-ca-configmap\") pod \"c1a77a15-50e0-43d2-b455-459f74de3250\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") "
Nov 02 01:49:34.847376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.847227    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a77a15-50e0-43d2-b455-459f74de3250-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a77a15-50e0-43d2-b455-459f74de3250-local-certs\") pod \"c1a77a15-50e0-43d2-b455-459f74de3250\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") "
Nov 02 01:49:34.847376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.847265    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-kube-api-access-sh4t8\" (UniqueName: \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-kube-api-access-sh4t8\") pod \"c1a77a15-50e0-43d2-b455-459f74de3250\" (UID: \"c1a77a15-50e0-43d2-b455-459f74de3250\") "
Nov 02 01:49:34.848101 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.848062    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" path="/var/lib/kubelet/pods/00fd0bde-f5d1-4a50-ad48-85bc6881cb27/volumes"
Nov 02 01:49:34.848700 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.848668    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-ca-configmap" pod "c1a77a15-50e0-43d2-b455-459f74de3250" (UID: "c1a77a15-50e0-43d2-b455-459f74de3250"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:34.850106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.850074    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" path="/var/lib/kubelet/pods/17113a57-21c8-46c1-a80a-deb91dc51a87/volumes"
Nov 02 01:49:34.851338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.851284    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-istio-token" pod "c1a77a15-50e0-43d2-b455-459f74de3250" (UID: "c1a77a15-50e0-43d2-b455-459f74de3250"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:34.853503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.853370    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-kubeconfig" pod "c1a77a15-50e0-43d2-b455-459f74de3250" (UID: "c1a77a15-50e0-43d2-b455-459f74de3250"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:34.853609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.853434    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-kube-api-access-sh4t8" pod "c1a77a15-50e0-43d2-b455-459f74de3250" (UID: "c1a77a15-50e0-43d2-b455-459f74de3250"). InnerVolumeSpecName "kube-api-access-sh4t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:34.854561 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.854539    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a77a15-50e0-43d2-b455-459f74de3250-local-certs" pod "c1a77a15-50e0-43d2-b455-459f74de3250" (UID: "c1a77a15-50e0-43d2-b455-459f74de3250"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:34.859956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.859929    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-dns-cert" pod "c1a77a15-50e0-43d2-b455-459f74de3250" (UID: "c1a77a15-50e0-43d2-b455-459f74de3250"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:34.860174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.860138    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-cacerts" pod "c1a77a15-50e0-43d2-b455-459f74de3250" (UID: "c1a77a15-50e0-43d2-b455-459f74de3250"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:34.860397 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.860373    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" path="/var/lib/kubelet/pods/ee158de2-d2d6-46dd-b0e8-b4e1a366633e/volumes"
Nov 02 01:49:34.948820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.948790    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh4t8\" (UniqueName: \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-kube-api-access-sh4t8\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.948820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.948818    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a77a15-50e0-43d2-b455-459f74de3250-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.948820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.948829    4851 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-cacerts\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.948820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.948839    4851 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-dns-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.949132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.948853    4851 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1a77a15-50e0-43d2-b455-459f74de3250-istio-kubeconfig\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.949132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.948865    4851 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1a77a15-50e0-43d2-b455-459f74de3250-istio-csr-ca-configmap\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:34.949132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:34.948874    4851 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a77a15-50e0-43d2-b455-459f74de3250-local-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:35.256350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.256319    4851 generic.go:356] "Generic (PLEG): container finished" podID="f66f9018-58ee-4d09-9ed7-0e20a4b6700c" containerID="5970461acdf68562a4ca5f189f90dbe0d10daae4314e661479e4dbc81b59380c" exitCode=0
Nov 02 01:49:35.256518 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.256386    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9" event={"ID":"f66f9018-58ee-4d09-9ed7-0e20a4b6700c","Type":"ContainerDied","Data":"5970461acdf68562a4ca5f189f90dbe0d10daae4314e661479e4dbc81b59380c"}
Nov 02 01:49:35.259893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.259871    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"
Nov 02 01:49:35.259893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.259883    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds" event={"ID":"bc860c6b-3659-4fce-b6ab-401e1264fc67","Type":"ContainerDied","Data":"180319cf1a8435acabb7022ed5054b530410c194c99b1f8ee319c55cb0d1156d"}
Nov 02 01:49:35.260134 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.259940    4851 scope.go:122] "RemoveContainer" containerID="a8f8284f26f5e5cf2a3a6c68a558b360b4cc5baff4d458bbe6edcf1842927494"
Nov 02 01:49:35.261524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.261503    4851 generic.go:356] "Generic (PLEG): container finished" podID="c1a77a15-50e0-43d2-b455-459f74de3250" containerID="6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555" exitCode=0
Nov 02 01:49:35.261639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.261594    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k" event={"ID":"c1a77a15-50e0-43d2-b455-459f74de3250","Type":"ContainerDied","Data":"6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555"}
Nov 02 01:49:35.261639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.261623    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"
Nov 02 01:49:35.261771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.261638    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k" event={"ID":"c1a77a15-50e0-43d2-b455-459f74de3250","Type":"ContainerDied","Data":"e56f9e456d291ba7cd2323b99b8b23bfdd6030330d1d7c41483e7e0744356dfb"}
Nov 02 01:49:35.272571 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.272538    4851 scope.go:122] "RemoveContainer" containerID="6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555"
Nov 02 01:49:35.285849 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.285813    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"]
Nov 02 01:49:35.289770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.289741    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-g9mds"]
Nov 02 01:49:35.293530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.293507    4851 scope.go:122] "RemoveContainer" containerID="6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555"
Nov 02 01:49:35.293930 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:35.293904    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555\": container with ID starting with 6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555 not found: ID does not exist" containerID="6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555"
Nov 02 01:49:35.294037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.293939    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555"} err="failed to get container status \"6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555\": rpc error: code = NotFound desc = could not find container \"6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555\": container with ID starting with 6e4ee9958ba6b41782c43ae635418efe0b6ab2eada2e0de78202ae9421166555 not found: ID does not exist"
Nov 02 01:49:35.308429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.305727    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"]
Nov 02 01:49:35.308429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.305778    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nnr4k"]
Nov 02 01:49:35.394600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.394576    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:49:35.441599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.441572    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:35.452692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.452664    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf5f50e-a1f8-43ac-9e17-163776479428-kube-api-access-j2zd2\" (UniqueName: \"kubernetes.io/projected/4cf5f50e-a1f8-43ac-9e17-163776479428-kube-api-access-j2zd2\") pod \"4cf5f50e-a1f8-43ac-9e17-163776479428\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") "
Nov 02 01:49:35.452872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.452699    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-kube-api-access-vzpq4\" (UniqueName: \"kubernetes.io/projected/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-kube-api-access-vzpq4\") pod \"f66f9018-58ee-4d09-9ed7-0e20a4b6700c\" (UID: \"f66f9018-58ee-4d09-9ed7-0e20a4b6700c\") "
Nov 02 01:49:35.452872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.452767    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-xks-ca\") pod \"4cf5f50e-a1f8-43ac-9e17-163776479428\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") "
Nov 02 01:49:35.452872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.452817    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-plugins-config-volume\") pod \"f66f9018-58ee-4d09-9ed7-0e20a4b6700c\" (UID: \"f66f9018-58ee-4d09-9ed7-0e20a4b6700c\") "
Nov 02 01:49:35.452872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.452846    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-openshift-ca\") pod \"4cf5f50e-a1f8-43ac-9e17-163776479428\" (UID: \"4cf5f50e-a1f8-43ac-9e17-163776479428\") "
Nov 02 01:49:35.453379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.453351    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-plugins-config-volume" pod "f66f9018-58ee-4d09-9ed7-0e20a4b6700c" (UID: "f66f9018-58ee-4d09-9ed7-0e20a4b6700c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:35.453563 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.453537    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-openshift-ca" pod "4cf5f50e-a1f8-43ac-9e17-163776479428" (UID: "4cf5f50e-a1f8-43ac-9e17-163776479428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:35.455483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.455454    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-xks-ca" pod "4cf5f50e-a1f8-43ac-9e17-163776479428" (UID: "4cf5f50e-a1f8-43ac-9e17-163776479428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:35.455613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.455591    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf5f50e-a1f8-43ac-9e17-163776479428-kube-api-access-j2zd2" pod "4cf5f50e-a1f8-43ac-9e17-163776479428" (UID: "4cf5f50e-a1f8-43ac-9e17-163776479428"). InnerVolumeSpecName "kube-api-access-j2zd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:35.455708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.455679    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-kube-api-access-vzpq4" pod "f66f9018-58ee-4d09-9ed7-0e20a4b6700c" (UID: "f66f9018-58ee-4d09-9ed7-0e20a4b6700c"). InnerVolumeSpecName "kube-api-access-vzpq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:35.554360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.554325    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2zd2\" (UniqueName: \"kubernetes.io/projected/4cf5f50e-a1f8-43ac-9e17-163776479428-kube-api-access-j2zd2\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:35.554360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.554352    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzpq4\" (UniqueName: \"kubernetes.io/projected/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-kube-api-access-vzpq4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:35.554360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.554362    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:35.554360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.554372    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66f9018-58ee-4d09-9ed7-0e20a4b6700c-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:35.554740 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:35.554380    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf5f50e-a1f8-43ac-9e17-163776479428-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:36.274844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.274805    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"
Nov 02 01:49:36.275019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.274816    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9" event={"ID":"f66f9018-58ee-4d09-9ed7-0e20a4b6700c","Type":"ContainerDied","Data":"1364f4c0296c96b7a86aae72dbd85b74080f59ce5befb9e427cfd7293ca2c792"}
Nov 02 01:49:36.275019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.274944    4851 scope.go:122] "RemoveContainer" containerID="5970461acdf68562a4ca5f189f90dbe0d10daae4314e661479e4dbc81b59380c"
Nov 02 01:49:36.287057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.286929    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"
Nov 02 01:49:36.287235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.287204    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf" event={"ID":"4cf5f50e-a1f8-43ac-9e17-163776479428","Type":"ContainerDied","Data":"936e89f2283485f2a84d66d8c88df14963dd6ca655025ad25eac8c0fece30ef7"}
Nov 02 01:49:36.287461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.287266    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="936e89f2283485f2a84d66d8c88df14963dd6ca655025ad25eac8c0fece30ef7"
Nov 02 01:49:36.303247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.303217    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"]
Nov 02 01:49:36.308623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.308591    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0f32d6-697d997987-km9s9"]
Nov 02 01:49:36.512437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.511460    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"]
Nov 02 01:49:36.516997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.516969    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ca1c53-nwfzc"]
Nov 02 01:49:36.846909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.846828    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c69e684-14de-4c9a-9166-3e4cacc890bc" path="/var/lib/kubelet/pods/4c69e684-14de-4c9a-9166-3e4cacc890bc/volumes"
Nov 02 01:49:36.847495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.847476    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" path="/var/lib/kubelet/pods/bc860c6b-3659-4fce-b6ab-401e1264fc67/volumes"
Nov 02 01:49:36.848582 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.848557    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a77a15-50e0-43d2-b455-459f74de3250" path="/var/lib/kubelet/pods/c1a77a15-50e0-43d2-b455-459f74de3250/volumes"
Nov 02 01:49:36.850629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.850601    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66f9018-58ee-4d09-9ed7-0e20a4b6700c" path="/var/lib/kubelet/pods/f66f9018-58ee-4d09-9ed7-0e20a4b6700c/volumes"
Nov 02 01:49:36.856085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:36.856051    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st" podUID="1e34ceb8-6e96-4404-bfd7-728e16738621" containerName="payload-pre-processing" containerID="cri-o://d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3" gracePeriod=25
Nov 02 01:49:37.624197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:37.624151    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st" podUID="1e34ceb8-6e96-4404-bfd7-728e16738621" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:38.633025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:38.632987    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8" podUID="c6ac98d6-c879-4606-81a4-b791e4af5b3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:39.696546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:39.696508    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln" podUID="c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:40.022653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:40.022556    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"]
Nov 02 01:49:40.025105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:40.025077    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f721a9-x7wl5"]
Nov 02 01:49:40.668918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:40.668883    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"]
Nov 02 01:49:40.669144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:40.669088    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy" containerID="cri-o://94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1" gracePeriod=30
Nov 02 01:49:40.845755 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:40.845714    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb611906-0a66-4656-8fa0-391c284c7c99" path="/var/lib/kubelet/pods/eb611906-0a66-4656-8fa0-391c284c7c99/volumes"
Nov 02 01:49:41.335061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.335017    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" containerID="cri-o://6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba" gracePeriod=45
Nov 02 01:49:41.508728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.508601    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": dial tcp 10.132.0.93:8000: connect: connection refused"
Nov 02 01:49:41.617268 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.617242    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:49:41.648239 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.648208    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-dshm\") pod \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") "
Nov 02 01:49:41.648472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.648269    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tls-certs\" (UniqueName: \"kubernetes.io/secret/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tls-certs\") pod \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") "
Nov 02 01:49:41.648472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.648330    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-home\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-home\") pod \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") "
Nov 02 01:49:41.648472 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.648380    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-model-cache\") pod \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") "
Nov 02 01:49:41.649072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.648710    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-home" pod "0af7a9a6-d01b-47cc-9762-0db8296f0b54" (UID: "0af7a9a6-d01b-47cc-9762-0db8296f0b54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:41.649072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.648711    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af7a9a6-d01b-47cc-9762-0db8296f0b54-kube-api-access-5w9h4\" (UniqueName: \"kubernetes.io/projected/0af7a9a6-d01b-47cc-9762-0db8296f0b54-kube-api-access-5w9h4\") pod \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") "
Nov 02 01:49:41.649072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.648790    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tmp-dir\") pod \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\" (UID: \"0af7a9a6-d01b-47cc-9762-0db8296f0b54\") "
Nov 02 01:49:41.649072 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.648945    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-model-cache" pod "0af7a9a6-d01b-47cc-9762-0db8296f0b54" (UID: "0af7a9a6-d01b-47cc-9762-0db8296f0b54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:41.649366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.649264    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:41.649366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.649203    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tmp-dir" pod "0af7a9a6-d01b-47cc-9762-0db8296f0b54" (UID: "0af7a9a6-d01b-47cc-9762-0db8296f0b54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:41.649366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.649284    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:41.651120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.651094    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af7a9a6-d01b-47cc-9762-0db8296f0b54-kube-api-access-5w9h4" pod "0af7a9a6-d01b-47cc-9762-0db8296f0b54" (UID: "0af7a9a6-d01b-47cc-9762-0db8296f0b54"). InnerVolumeSpecName "kube-api-access-5w9h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:41.651502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.651476    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tls-certs" pod "0af7a9a6-d01b-47cc-9762-0db8296f0b54" (UID: "0af7a9a6-d01b-47cc-9762-0db8296f0b54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:41.651631 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.651611    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-dshm" pod "0af7a9a6-d01b-47cc-9762-0db8296f0b54" (UID: "0af7a9a6-d01b-47cc-9762-0db8296f0b54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:41.750975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.750935    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:41.750975 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.750977    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af7a9a6-d01b-47cc-9762-0db8296f0b54-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:41.751185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.750991    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af7a9a6-d01b-47cc-9762-0db8296f0b54-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:41.751185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:41.751007    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9h4\" (UniqueName: \"kubernetes.io/projected/0af7a9a6-d01b-47cc-9762-0db8296f0b54-kube-api-access-5w9h4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:42.352566 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:42.352532    4851 generic.go:356] "Generic (PLEG): container finished" podID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerID="6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba" exitCode=0
Nov 02 01:49:42.353019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:42.352578    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"
Nov 02 01:49:42.353019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:42.352612    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" event={"ID":"0af7a9a6-d01b-47cc-9762-0db8296f0b54","Type":"ContainerDied","Data":"6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba"}
Nov 02 01:49:42.353019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:42.352652    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p" event={"ID":"0af7a9a6-d01b-47cc-9762-0db8296f0b54","Type":"ContainerDied","Data":"9e596dc00c814f8a9afc47e1da0c063ece957368cacc27ea0499e33aed70c2f5"}
Nov 02 01:49:42.353019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:42.352668    4851 scope.go:122] "RemoveContainer" containerID="6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba"
Nov 02 01:49:42.364309 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:42.364275    4851 scope.go:122] "RemoveContainer" containerID="6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba"
Nov 02 01:49:42.364652 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:42.364627    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba\": container with ID starting with 6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba not found: ID does not exist" containerID="6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba"
Nov 02 01:49:42.364696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:42.364664    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba"} err="failed to get container status \"6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba\": rpc error: code = NotFound desc = could not find container \"6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba\": container with ID starting with 6f0cc1ed2496fc5a7d4ca7872202d673391e8e166bb7d401fbbe9bdff4aa9bba not found: ID does not exist"
Nov 02 01:49:42.376439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:42.376404    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"]
Nov 02 01:49:42.378882 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:42.378859    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-2-w5-123986-kserve-5f5c86db8b-vnj4p"]
Nov 02 01:49:42.844896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:42.844860    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" path="/var/lib/kubelet/pods/0af7a9a6-d01b-47cc-9762-0db8296f0b54/volumes"
Nov 02 01:49:43.673905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:43.673868    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"]
Nov 02 01:49:43.674468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:43.674138    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy" containerID="cri-o://ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb" gracePeriod=30
Nov 02 01:49:45.932285 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:45.932250    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:49:45.995656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:45.995582    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-data\") pod \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") "
Nov 02 01:49:45.995656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:45.995624    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-podinfo\") pod \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") "
Nov 02 01:49:45.995656 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:45.995650    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-certs\") pod \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") "
Nov 02 01:49:45.995894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:45.995690    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-credential-socket\") pod \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") "
Nov 02 01:49:45.995894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:45.995714    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istiod-ca-cert\") pod \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") "
Nov 02 01:49:45.995894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:45.995771    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-envoy\") pod \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") "
Nov 02 01:49:45.995894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:45.995788    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-token\" (UniqueName: \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-token\") pod \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") "
Nov 02 01:49:45.995894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:45.995826    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-socket\") pod \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") "
Nov 02 01:49:45.995894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:45.995860    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-kube-api-access-lltc7\" (UniqueName: \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-kube-api-access-lltc7\") pod \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\" (UID: \"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd\") "
Nov 02 01:49:46.033159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.033118    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-certs" pod "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" (UID: "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:46.033357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.033172    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-data" pod "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" (UID: "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:46.033640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.033621    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-kube-api-access-lltc7" pod "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" (UID: "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd"). InnerVolumeSpecName "kube-api-access-lltc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:46.034092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.034072    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-envoy" pod "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" (UID: "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:46.034482 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.034460    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-podinfo" pod "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" (UID: "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:49:46.034934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.034894    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-token" pod "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" (UID: "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:46.035414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.035393    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istiod-ca-cert" pod "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" (UID: "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:46.035783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.035769    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-credential-socket" pod "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" (UID: "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:46.035783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.035775    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-socket" pod "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" (UID: "4e8e15b0-c38a-471b-b7e5-3cb8987f49dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:46.096719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.096687    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:46.096719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.096716    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:46.096719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.096726    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:46.096949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.096734    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:46.096949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.096743    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:46.096949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.096753    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:46.096949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.096762    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:46.096949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.096771    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lltc7\" (UniqueName: \"kubernetes.io/projected/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-kube-api-access-lltc7\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:46.096949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.096780    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:46.383878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.383791    4851 generic.go:356] "Generic (PLEG): container finished" podID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerID="94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1" exitCode=0
Nov 02 01:49:46.383878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.383829    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" event={"ID":"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd","Type":"ContainerDied","Data":"94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1"}
Nov 02 01:49:46.383878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.383839    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"
Nov 02 01:49:46.383878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.383872    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" event={"ID":"4e8e15b0-c38a-471b-b7e5-3cb8987f49dd","Type":"ContainerDied","Data":"b06b37e325451ed9baedb6d566468d8b94f922d8a3e05d3476144cd7e9523bb0"}
Nov 02 01:49:46.384191 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.383894    4851 scope.go:122] "RemoveContainer" containerID="94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1"
Nov 02 01:49:46.399076 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.399049    4851 scope.go:122] "RemoveContainer" containerID="94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1"
Nov 02 01:49:46.399429 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:46.399404    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1\": container with ID starting with 94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1 not found: ID does not exist" containerID="94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1"
Nov 02 01:49:46.399510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.399438    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1"} err="failed to get container status \"94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1\": rpc error: code = NotFound desc = could not find container \"94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1\": container with ID starting with 94f57605e4cda44609238bf57e0472fa0cbf1f656a29ef9464591e62a291a0c1 not found: ID does not exist"
Nov 02 01:49:46.409925 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.409899    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"]
Nov 02 01:49:46.417275 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.417252    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h"]
Nov 02 01:49:46.846605 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.846575    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" path="/var/lib/kubelet/pods/4e8e15b0-c38a-471b-b7e5-3cb8987f49dd/volumes"
Nov 02 01:49:46.878065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.878014    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7f78h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:46.878271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:46.878080    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7f78h" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:47.096687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:47.096586    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"]
Nov 02 01:49:47.106924 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:47.106896    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-123986-jg6sr"]
Nov 02 01:49:47.624850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:47.624819    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st" podUID="1e34ceb8-6e96-4404-bfd7-728e16738621" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:49:47.663728 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:47.663683    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:49:48.632743 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:49:48.632714    4851 logging.go:55] [core] [Channel #6723 SubChannel #6724]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.51:50052", ServerName: "10.132.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.51:50052: connect: connection refused"
Nov 02 01:49:48.809105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:48.809059    4851 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 01:49:48.809253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:48.809174    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 01:49:48.853196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:48.851116    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="053d5001-a748-4c58-a682-dafb91cacbb3" path="/var/lib/kubelet/pods/053d5001-a748-4c58-a682-dafb91cacbb3/volumes"
Nov 02 01:49:48.959568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:48.959537    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:49:49.001199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.001177    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:49:49.006529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.006504    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:49:49.024384 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.024357    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-kube-api-access-4bfz2\" (UniqueName: \"kubernetes.io/projected/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-kube-api-access-4bfz2\") pod \"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe\" (UID: \"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe\") "
Nov 02 01:49:49.024615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.024442    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ac98d6-c879-4606-81a4-b791e4af5b3f-kube-api-access-6pvhk\" (UniqueName: \"kubernetes.io/projected/c6ac98d6-c879-4606-81a4-b791e4af5b3f-kube-api-access-6pvhk\") pod \"c6ac98d6-c879-4606-81a4-b791e4af5b3f\" (UID: \"c6ac98d6-c879-4606-81a4-b791e4af5b3f\") "
Nov 02 01:49:49.024615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.024533    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-plugins-config-volume\") pod \"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe\" (UID: \"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe\") "
Nov 02 01:49:49.024615 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.024592    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ac98d6-c879-4606-81a4-b791e4af5b3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ac98d6-c879-4606-81a4-b791e4af5b3f-plugins-config-volume\") pod \"c6ac98d6-c879-4606-81a4-b791e4af5b3f\" (UID: \"c6ac98d6-c879-4606-81a4-b791e4af5b3f\") "
Nov 02 01:49:49.025369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.025342    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-plugins-config-volume" pod "c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" (UID: "c10e0dc1-0c5b-4bc6-bb28-06a129774bfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:49.025892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.025861    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ac98d6-c879-4606-81a4-b791e4af5b3f-plugins-config-volume" pod "c6ac98d6-c879-4606-81a4-b791e4af5b3f" (UID: "c6ac98d6-c879-4606-81a4-b791e4af5b3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:49.029083 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.029059    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ac98d6-c879-4606-81a4-b791e4af5b3f-kube-api-access-6pvhk" pod "c6ac98d6-c879-4606-81a4-b791e4af5b3f" (UID: "c6ac98d6-c879-4606-81a4-b791e4af5b3f"). InnerVolumeSpecName "kube-api-access-6pvhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:49.029489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.029467    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-kube-api-access-4bfz2" pod "c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" (UID: "c10e0dc1-0c5b-4bc6-bb28-06a129774bfe"). InnerVolumeSpecName "kube-api-access-4bfz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:49.126139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126098    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-podinfo\") pod \"a3f759bd-f068-415d-b56f-54c4e7a7545a\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") "
Nov 02 01:49:49.126139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126147    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-certs\") pod \"a3f759bd-f068-415d-b56f-54c4e7a7545a\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") "
Nov 02 01:49:49.126444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126186    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-credential-socket\") pod \"a3f759bd-f068-415d-b56f-54c4e7a7545a\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") "
Nov 02 01:49:49.126444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126206    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-kube-api-access-4fnlv\" (UniqueName: \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-kube-api-access-4fnlv\") pod \"a3f759bd-f068-415d-b56f-54c4e7a7545a\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") "
Nov 02 01:49:49.126444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126324    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f759bd-f068-415d-b56f-54c4e7a7545a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f759bd-f068-415d-b56f-54c4e7a7545a-istiod-ca-cert\") pod \"a3f759bd-f068-415d-b56f-54c4e7a7545a\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") "
Nov 02 01:49:49.126444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126370    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-data\") pod \"a3f759bd-f068-415d-b56f-54c4e7a7545a\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") "
Nov 02 01:49:49.126444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126398    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-token\" (UniqueName: \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-token\") pod \"a3f759bd-f068-415d-b56f-54c4e7a7545a\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") "
Nov 02 01:49:49.126810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126451    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-socket\") pod \"a3f759bd-f068-415d-b56f-54c4e7a7545a\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") "
Nov 02 01:49:49.126810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126509    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-envoy\") pod \"a3f759bd-f068-415d-b56f-54c4e7a7545a\" (UID: \"a3f759bd-f068-415d-b56f-54c4e7a7545a\") "
Nov 02 01:49:49.126810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126628    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-certs" pod "a3f759bd-f068-415d-b56f-54c4e7a7545a" (UID: "a3f759bd-f068-415d-b56f-54c4e7a7545a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:49.126810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126647    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-credential-socket" pod "a3f759bd-f068-415d-b56f-54c4e7a7545a" (UID: "a3f759bd-f068-415d-b56f-54c4e7a7545a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:49.126810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126742    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-data" pod "a3f759bd-f068-415d-b56f-54c4e7a7545a" (UID: "a3f759bd-f068-415d-b56f-54c4e7a7545a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:49.127266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126924    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-socket" pod "a3f759bd-f068-415d-b56f-54c4e7a7545a" (UID: "a3f759bd-f068-415d-b56f-54c4e7a7545a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:49.127266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126969    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.127266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.126993    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.127266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.127010    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ac98d6-c879-4606-81a4-b791e4af5b3f-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.127266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.127025    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bfz2\" (UniqueName: \"kubernetes.io/projected/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe-kube-api-access-4bfz2\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.127266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.127038    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.127266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.127052    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.127266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.127061    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f759bd-f068-415d-b56f-54c4e7a7545a-istiod-ca-cert" pod "a3f759bd-f068-415d-b56f-54c4e7a7545a" (UID: "a3f759bd-f068-415d-b56f-54c4e7a7545a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:49.127266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.127066    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pvhk\" (UniqueName: \"kubernetes.io/projected/c6ac98d6-c879-4606-81a4-b791e4af5b3f-kube-api-access-6pvhk\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.128741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.128711    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-podinfo" pod "a3f759bd-f068-415d-b56f-54c4e7a7545a" (UID: "a3f759bd-f068-415d-b56f-54c4e7a7545a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:49:49.128857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.128772    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-kube-api-access-4fnlv" pod "a3f759bd-f068-415d-b56f-54c4e7a7545a" (UID: "a3f759bd-f068-415d-b56f-54c4e7a7545a"). InnerVolumeSpecName "kube-api-access-4fnlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:49.128918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.128868    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-envoy" pod "a3f759bd-f068-415d-b56f-54c4e7a7545a" (UID: "a3f759bd-f068-415d-b56f-54c4e7a7545a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:49.129020 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.128997    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-token" pod "a3f759bd-f068-415d-b56f-54c4e7a7545a" (UID: "a3f759bd-f068-415d-b56f-54c4e7a7545a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:49.227776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.227695    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fnlv\" (UniqueName: \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-kube-api-access-4fnlv\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.227776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.227723    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f759bd-f068-415d-b56f-54c4e7a7545a-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.227776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.227733    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.227776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.227742    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.227776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.227750    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.227776 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.227759    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f759bd-f068-415d-b56f-54c4e7a7545a-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:49.411018 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.410980    4851 generic.go:356] "Generic (PLEG): container finished" podID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerID="ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb" exitCode=0
Nov 02 01:49:49.411207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.411030    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"
Nov 02 01:49:49.411207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.411080    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" event={"ID":"a3f759bd-f068-415d-b56f-54c4e7a7545a","Type":"ContainerDied","Data":"ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb"}
Nov 02 01:49:49.411207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.411134    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7" event={"ID":"a3f759bd-f068-415d-b56f-54c4e7a7545a","Type":"ContainerDied","Data":"5a35740a98b91dbdad0d8aff49bf8f1eabc0044d464297d7e1b5fa46ec5d3bca"}
Nov 02 01:49:49.411207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.411159    4851 scope.go:122] "RemoveContainer" containerID="ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb"
Nov 02 01:49:49.412695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.412672    4851 generic.go:356] "Generic (PLEG): container finished" podID="c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" containerID="281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b" exitCode=0
Nov 02 01:49:49.412825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.412718    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"
Nov 02 01:49:49.412825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.412786    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln" event={"ID":"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe","Type":"ContainerDied","Data":"281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b"}
Nov 02 01:49:49.412937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.412839    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln" event={"ID":"c10e0dc1-0c5b-4bc6-bb28-06a129774bfe","Type":"ContainerDied","Data":"be0ee497134abc716ea74df66cec3ded4b6ee39253f8f0394d268a88e1e44157"}
Nov 02 01:49:49.414420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.414397    4851 generic.go:356] "Generic (PLEG): container finished" podID="c6ac98d6-c879-4606-81a4-b791e4af5b3f" containerID="54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d" exitCode=0
Nov 02 01:49:49.414542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.414441    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"
Nov 02 01:49:49.414542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.414447    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8" event={"ID":"c6ac98d6-c879-4606-81a4-b791e4af5b3f","Type":"ContainerDied","Data":"54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d"}
Nov 02 01:49:49.414542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.414485    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8" event={"ID":"c6ac98d6-c879-4606-81a4-b791e4af5b3f","Type":"ContainerDied","Data":"62fa6380c31f500f29b99e1a572cde9cf85f9ce3444d0eb85e2d46f55dc13085"}
Nov 02 01:49:49.422241 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.422223    4851 scope.go:122] "RemoveContainer" containerID="ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb"
Nov 02 01:49:49.422524 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:49.422494    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb\": container with ID starting with ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb not found: ID does not exist" containerID="ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb"
Nov 02 01:49:49.422632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.422536    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb"} err="failed to get container status \"ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb\": rpc error: code = NotFound desc = could not find container \"ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb\": container with ID starting with ecb528cc76686c93c7e6db7c8500bf1f92e32511bda907451f63ca5676bc4bfb not found: ID does not exist"
Nov 02 01:49:49.422632 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.422559    4851 scope.go:122] "RemoveContainer" containerID="281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b"
Nov 02 01:49:49.432893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.432874    4851 scope.go:122] "RemoveContainer" containerID="281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b"
Nov 02 01:49:49.433151 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:49.433133    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b\": container with ID starting with 281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b not found: ID does not exist" containerID="281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b"
Nov 02 01:49:49.433199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.433160    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b"} err="failed to get container status \"281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b\": rpc error: code = NotFound desc = could not find container \"281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b\": container with ID starting with 281f0830787a808440a210339de02bad39f76e6522200de1707c47262d65d54b not found: ID does not exist"
Nov 02 01:49:49.433199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.433175    4851 scope.go:122] "RemoveContainer" containerID="54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d"
Nov 02 01:49:49.442387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.442365    4851 scope.go:122] "RemoveContainer" containerID="54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d"
Nov 02 01:49:49.442657 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:49.442637    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d\": container with ID starting with 54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d not found: ID does not exist" containerID="54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d"
Nov 02 01:49:49.442719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.442665    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d"} err="failed to get container status \"54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d\": rpc error: code = NotFound desc = could not find container \"54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d\": container with ID starting with 54e6081e555b11ae1c7774f55a31565f0094366baa65160acc942b291ee2333d not found: ID does not exist"
Nov 02 01:49:49.447544 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.447511    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"]
Nov 02 01:49:49.463235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.463215    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ca1c53-7b6cf54c75-78qln"]
Nov 02 01:49:49.493864 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.493800    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"]
Nov 02 01:49:49.508698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.508675    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-4mfw7"]
Nov 02 01:49:49.546413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.546384    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"]
Nov 02 01:49:49.550958 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.550920    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8"]
Nov 02 01:49:49.632553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:49.632524    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ca1c53-7c7c96c8b7-gqtq8" podUID="c6ac98d6-c879-4606-81a4-b791e4af5b3f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.51:50052\" within 1s: context deadline exceeded"
Nov 02 01:49:50.844571 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:50.844537    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" path="/var/lib/kubelet/pods/a3f759bd-f068-415d-b56f-54c4e7a7545a/volumes"
Nov 02 01:49:50.844950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:50.844897    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" path="/var/lib/kubelet/pods/c10e0dc1-0c5b-4bc6-bb28-06a129774bfe/volumes"
Nov 02 01:49:50.845219 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:50.845207    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ac98d6-c879-4606-81a4-b791e4af5b3f" path="/var/lib/kubelet/pods/c6ac98d6-c879-4606-81a4-b791e4af5b3f/volumes"
Nov 02 01:49:53.876487 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:53.876451    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"]
Nov 02 01:49:53.877085 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:53.876781    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy" containerID="cri-o://c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9" gracePeriod=30
Nov 02 01:49:56.387314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:56.387256    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:49:56.388350 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:56.388329    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:49:57.116046 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.115973    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:49:57.203797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.203760    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e34ceb8-6e96-4404-bfd7-728e16738621-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e34ceb8-6e96-4404-bfd7-728e16738621-plugins-config-volume\") pod \"1e34ceb8-6e96-4404-bfd7-728e16738621\" (UID: \"1e34ceb8-6e96-4404-bfd7-728e16738621\") "
Nov 02 01:49:57.204030 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.203860    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e34ceb8-6e96-4404-bfd7-728e16738621-kube-api-access-bssbh\" (UniqueName: \"kubernetes.io/projected/1e34ceb8-6e96-4404-bfd7-728e16738621-kube-api-access-bssbh\") pod \"1e34ceb8-6e96-4404-bfd7-728e16738621\" (UID: \"1e34ceb8-6e96-4404-bfd7-728e16738621\") "
Nov 02 01:49:57.204188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.204163    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e34ceb8-6e96-4404-bfd7-728e16738621-plugins-config-volume" pod "1e34ceb8-6e96-4404-bfd7-728e16738621" (UID: "1e34ceb8-6e96-4404-bfd7-728e16738621"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:57.206112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.206091    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e34ceb8-6e96-4404-bfd7-728e16738621-kube-api-access-bssbh" pod "1e34ceb8-6e96-4404-bfd7-728e16738621" (UID: "1e34ceb8-6e96-4404-bfd7-728e16738621"). InnerVolumeSpecName "kube-api-access-bssbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:57.304902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.304860    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bssbh\" (UniqueName: \"kubernetes.io/projected/1e34ceb8-6e96-4404-bfd7-728e16738621-kube-api-access-bssbh\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:57.304902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.304896    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e34ceb8-6e96-4404-bfd7-728e16738621-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:57.483368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.483335    4851 generic.go:356] "Generic (PLEG): container finished" podID="1e34ceb8-6e96-4404-bfd7-728e16738621" containerID="d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3" exitCode=0
Nov 02 01:49:57.483890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.483398    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"
Nov 02 01:49:57.483890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.483432    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st" event={"ID":"1e34ceb8-6e96-4404-bfd7-728e16738621","Type":"ContainerDied","Data":"d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3"}
Nov 02 01:49:57.483890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.483491    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st" event={"ID":"1e34ceb8-6e96-4404-bfd7-728e16738621","Type":"ContainerDied","Data":"b3660ae04552b2d965e77370bc89a7bc1fd3390637941143537528259dd994fa"}
Nov 02 01:49:57.483890 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.483513    4851 scope.go:122] "RemoveContainer" containerID="d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3"
Nov 02 01:49:57.496232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.496208    4851 scope.go:122] "RemoveContainer" containerID="d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3"
Nov 02 01:49:57.496584 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:57.496565    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3\": container with ID starting with d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3 not found: ID does not exist" containerID="d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3"
Nov 02 01:49:57.496637 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.496595    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3"} err="failed to get container status \"d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3\": rpc error: code = NotFound desc = could not find container \"d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3\": container with ID starting with d366c1a99dc14bbe0af282323a6ca313763aafcfce8d1d52559972341f735ca3 not found: ID does not exist"
Nov 02 01:49:57.514413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.510073    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"]
Nov 02 01:49:57.514413 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:57.513922    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-123986-68cdf4b4c5-dh7st"]
Nov 02 01:49:58.846270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:58.846230    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e34ceb8-6e96-4404-bfd7-728e16738621" path="/var/lib/kubelet/pods/1e34ceb8-6e96-4404-bfd7-728e16738621/volumes"
Nov 02 01:49:59.212629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.212604    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:49:59.328138 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328087    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-socket\") pod \"39c0169e-44d5-4070-83f8-fbf617820e95\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") "
Nov 02 01:49:59.328378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328189    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-istio-token\" (UniqueName: \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-istio-token\") pod \"39c0169e-44d5-4070-83f8-fbf617820e95\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") "
Nov 02 01:49:59.328378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328213    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c0169e-44d5-4070-83f8-fbf617820e95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c0169e-44d5-4070-83f8-fbf617820e95-istiod-ca-cert\") pod \"39c0169e-44d5-4070-83f8-fbf617820e95\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") "
Nov 02 01:49:59.328378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328254    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-data\") pod \"39c0169e-44d5-4070-83f8-fbf617820e95\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") "
Nov 02 01:49:59.328378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328282    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-envoy\") pod \"39c0169e-44d5-4070-83f8-fbf617820e95\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") "
Nov 02 01:49:59.328378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328353    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-kube-api-access-khgrv\" (UniqueName: \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-kube-api-access-khgrv\") pod \"39c0169e-44d5-4070-83f8-fbf617820e95\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") "
Nov 02 01:49:59.328697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328395    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-certs\") pod \"39c0169e-44d5-4070-83f8-fbf617820e95\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") "
Nov 02 01:49:59.328697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328463    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-credential-socket\") pod \"39c0169e-44d5-4070-83f8-fbf617820e95\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") "
Nov 02 01:49:59.328697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328472    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-socket" pod "39c0169e-44d5-4070-83f8-fbf617820e95" (UID: "39c0169e-44d5-4070-83f8-fbf617820e95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:59.328697 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328517    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39c0169e-44d5-4070-83f8-fbf617820e95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c0169e-44d5-4070-83f8-fbf617820e95-istio-podinfo\") pod \"39c0169e-44d5-4070-83f8-fbf617820e95\" (UID: \"39c0169e-44d5-4070-83f8-fbf617820e95\") "
Nov 02 01:49:59.328919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328691    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-data" pod "39c0169e-44d5-4070-83f8-fbf617820e95" (UID: "39c0169e-44d5-4070-83f8-fbf617820e95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:59.328919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328765    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-certs" pod "39c0169e-44d5-4070-83f8-fbf617820e95" (UID: "39c0169e-44d5-4070-83f8-fbf617820e95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:59.328919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328867    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-credential-socket" pod "39c0169e-44d5-4070-83f8-fbf617820e95" (UID: "39c0169e-44d5-4070-83f8-fbf617820e95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:59.328919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328901    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:59.328919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328920    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:59.329154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.328934    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:59.329154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.329070    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c0169e-44d5-4070-83f8-fbf617820e95-istiod-ca-cert" pod "39c0169e-44d5-4070-83f8-fbf617820e95" (UID: "39c0169e-44d5-4070-83f8-fbf617820e95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:59.331195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.331173    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-envoy" pod "39c0169e-44d5-4070-83f8-fbf617820e95" (UID: "39c0169e-44d5-4070-83f8-fbf617820e95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:49:59.331195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.331175    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-kube-api-access-khgrv" pod "39c0169e-44d5-4070-83f8-fbf617820e95" (UID: "39c0169e-44d5-4070-83f8-fbf617820e95"). InnerVolumeSpecName "kube-api-access-khgrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:59.331195 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.331184    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39c0169e-44d5-4070-83f8-fbf617820e95-istio-podinfo" pod "39c0169e-44d5-4070-83f8-fbf617820e95" (UID: "39c0169e-44d5-4070-83f8-fbf617820e95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:49:59.331410 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.331193    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-istio-token" pod "39c0169e-44d5-4070-83f8-fbf617820e95" (UID: "39c0169e-44d5-4070-83f8-fbf617820e95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:59.429676 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.429624    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:59.429676 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.429673    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c0169e-44d5-4070-83f8-fbf617820e95-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:59.429676 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.429682    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:59.429940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.429701    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c0169e-44d5-4070-83f8-fbf617820e95-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:59.429940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.429732    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c0169e-44d5-4070-83f8-fbf617820e95-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:59.429940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.429746    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khgrv\" (UniqueName: \"kubernetes.io/projected/39c0169e-44d5-4070-83f8-fbf617820e95-kube-api-access-khgrv\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:49:59.503961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.503873    4851 generic.go:356] "Generic (PLEG): container finished" podID="39c0169e-44d5-4070-83f8-fbf617820e95" containerID="c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9" exitCode=0
Nov 02 01:49:59.503961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.503921    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"
Nov 02 01:49:59.503961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.503944    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" event={"ID":"39c0169e-44d5-4070-83f8-fbf617820e95","Type":"ContainerDied","Data":"c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9"}
Nov 02 01:49:59.504255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.503986    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b" event={"ID":"39c0169e-44d5-4070-83f8-fbf617820e95","Type":"ContainerDied","Data":"2498c180159843fe0a5ef9583f267bd7779cade46a43bf2a7c7574459e6c3624"}
Nov 02 01:49:59.504255 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.504008    4851 scope.go:122] "RemoveContainer" containerID="c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9"
Nov 02 01:49:59.515589 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.515562    4851 scope.go:122] "RemoveContainer" containerID="c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9"
Nov 02 01:49:59.515901 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:49:59.515877    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9\": container with ID starting with c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9 not found: ID does not exist" containerID="c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9"
Nov 02 01:49:59.515985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.515915    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9"} err="failed to get container status \"c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9\": rpc error: code = NotFound desc = could not find container \"c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9\": container with ID starting with c4b4964e491b8f25a00c8cbb0190df750e83cef0d8ccbc9155a78f9ee02375c9 not found: ID does not exist"
Nov 02 01:49:59.526886 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.526864    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"]
Nov 02 01:49:59.531524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:49:59.531497    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-qxm8b"]
Nov 02 01:50:00.845079 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:00.845031    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" path="/var/lib/kubelet/pods/39c0169e-44d5-4070-83f8-fbf617820e95/volumes"
Nov 02 01:50:02.022027 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.021994    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"]
Nov 02 01:50:02.024915 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.024891    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd4962e9-9hqh9"]
Nov 02 01:50:02.845418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.845383    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96ed607-e467-4e00-86af-ac6f44a46e54" path="/var/lib/kubelet/pods/b96ed607-e467-4e00-86af-ac6f44a46e54/volumes"
Nov 02 01:50:02.876317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.876272    4851 scope.go:122] "RemoveContainer" containerID="8eb06e096838557dba9a22edf20c36f35f2b5984e2634e6ad32f69810dbdacb4"
Nov 02 01:50:02.887835 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.887807    4851 scope.go:122] "RemoveContainer" containerID="ff547042f13b571884c94b2901c7961ce487b7ddfea54f5bda51554fc2a61cf4"
Nov 02 01:50:02.896910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.896892    4851 scope.go:122] "RemoveContainer" containerID="e93ec1f5bd8fd2930209b7965ffcc4f93d7ff431f89d4a2a1e52d4ce7e51274a"
Nov 02 01:50:02.905879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.905860    4851 scope.go:122] "RemoveContainer" containerID="58aab98182e964349ad1a30c68d6d80218405f84683d5a26efebfffb956db4aa"
Nov 02 01:50:02.914770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.914750    4851 scope.go:122] "RemoveContainer" containerID="66658adc13a26d5c359172f6782656988e7617966c238f2bcea78ddf66a7d6f9"
Nov 02 01:50:02.926709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.926673    4851 scope.go:122] "RemoveContainer" containerID="70f64862bc10c8934cce38c3dd2b0553378a5f242678f5c7ebe446aab5ea0b62"
Nov 02 01:50:02.938937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.938910    4851 scope.go:122] "RemoveContainer" containerID="ee61117e1dbc885797c231651d6f26885dbdf83664ff1c1084de0433751e32a9"
Nov 02 01:50:02.947955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.947932    4851 scope.go:122] "RemoveContainer" containerID="821f3e968476e466723c96e060839604f4c81b4aefc8d79af189c127c8ba7e18"
Nov 02 01:50:02.957676 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.957657    4851 scope.go:122] "RemoveContainer" containerID="f49d54d55ea1221afff9258e90a337fb2ffd8426e1c81a2f828a04460c999153"
Nov 02 01:50:02.966678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.966657    4851 scope.go:122] "RemoveContainer" containerID="f69ba84197c25f21e3a608013a9a93b56894f9f5355d38b781388c776edb7ca9"
Nov 02 01:50:02.975155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:02.975126    4851 scope.go:122] "RemoveContainer" containerID="aa81fedd6741cf555cff47627fd16b7b88238a8bac77b251917fc405252c8310"
Nov 02 01:50:13.033797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:13.033680    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"]
Nov 02 01:50:13.036244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:13.036215    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009155c0-v8sr6"]
Nov 02 01:50:14.848774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:14.848732    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b544bc-e8ac-429d-aa90-452a2e294d16" path="/var/lib/kubelet/pods/56b544bc-e8ac-429d-aa90-452a2e294d16/volumes"
Nov 02 01:50:17.593777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.593743    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"]
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594107    4851 cpu_manager.go:414] "containerMap: removing container" podUID="1e34ceb8-6e96-4404-bfd7-728e16738621" containerName="payload-pre-processing"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594118    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e34ceb8-6e96-4404-bfd7-728e16738621" containerName="payload-pre-processing"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594127    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c6ac98d6-c879-4606-81a4-b791e4af5b3f" containerName="payload-pre-processing"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594132    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ac98d6-c879-4606-81a4-b791e4af5b3f" containerName="payload-pre-processing"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594144    4851 cpu_manager.go:414] "containerMap: removing container" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594149    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594160    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594165    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594173    4851 cpu_manager.go:414] "containerMap: removing container" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594177    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594187    4851 cpu_manager.go:414] "containerMap: removing container" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594191    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594198    4851 cpu_manager.go:414] "containerMap: removing container" podUID="4cf5f50e-a1f8-43ac-9e17-163776479428" containerName="revoke-keys"
Nov 02 01:50:17.594199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594202    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf5f50e-a1f8-43ac-9e17-163776479428" containerName="revoke-keys"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594210    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" containerName="payload-processing"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594214    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" containerName="payload-processing"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594219    4851 cpu_manager.go:414] "containerMap: removing container" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594223    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594237    4851 cpu_manager.go:414] "containerMap: removing container" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594241    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594248    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594252    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594259    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c1a77a15-50e0-43d2-b455-459f74de3250" containerName="discovery"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594263    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a77a15-50e0-43d2-b455-459f74de3250" containerName="discovery"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594269    4851 cpu_manager.go:414] "containerMap: removing container" podUID="f66f9018-58ee-4d09-9ed7-0e20a4b6700c" containerName="payload-processing"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594274    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66f9018-58ee-4d09-9ed7-0e20a4b6700c" containerName="payload-processing"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594283    4851 cpu_manager.go:414] "containerMap: removing container" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594301    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594309    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594316    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594373    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8e15b0-c38a-471b-b7e5-3cb8987f49dd" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594383    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d486ac0b-a945-45aa-b47b-9ece667cb1f1" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594389    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a77a15-50e0-43d2-b455-459f74de3250" containerName="discovery"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594395    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66f9018-58ee-4d09-9ed7-0e20a4b6700c" containerName="payload-processing"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594401    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c0169e-44d5-4070-83f8-fbf617820e95" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594407    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af7a9a6-d01b-47cc-9762-0db8296f0b54" containerName="main"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594413    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e34ceb8-6e96-4404-bfd7-728e16738621" containerName="payload-pre-processing"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594419    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ac98d6-c879-4606-81a4-b791e4af5b3f" containerName="payload-pre-processing"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594425    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee158de2-d2d6-46dd-b0e8-b4e1a366633e" containerName="main"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594431    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc860c6b-3659-4fce-b6ab-401e1264fc67" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594436    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="17113a57-21c8-46c1-a80a-deb91dc51a87" containerName="main"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594442    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f759bd-f068-415d-b56f-54c4e7a7545a" containerName="istio-proxy"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594447    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf5f50e-a1f8-43ac-9e17-163776479428" containerName="revoke-keys"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594453    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10e0dc1-0c5b-4bc6-bb28-06a129774bfe" containerName="payload-processing"
Nov 02 01:50:17.594679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.594458    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fd0bde-f5d1-4a50-ad48-85bc6881cb27" containerName="main"
Nov 02 01:50:17.601465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.601448    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:17.606217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.606190    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:50:17.606674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.606277    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:50:17.606674 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.606281    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:50:17.607060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.607036    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"]
Nov 02 01:50:17.607441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.607410    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:50:17.607558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.607445    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbk8\" (UniqueName: \"kubernetes.io/projected/769e37b8-624f-492c-9f56-dbd965280659-kube-api-access-4cbk8\") pod \"maas-api-revoke-sub-85237e45-kj96s\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:17.607558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.607551    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769e37b8-624f-492c-9f56-dbd965280659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kj96s\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:17.607698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.607615    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769e37b8-624f-492c-9f56-dbd965280659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kj96s\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:17.656237 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:50:17.656200    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:50:17.708758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.708719    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbk8\" (UniqueName: \"kubernetes.io/projected/769e37b8-624f-492c-9f56-dbd965280659-kube-api-access-4cbk8\") pod \"maas-api-revoke-sub-85237e45-kj96s\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:17.708947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.708771    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769e37b8-624f-492c-9f56-dbd965280659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kj96s\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:17.708947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.708820    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769e37b8-624f-492c-9f56-dbd965280659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kj96s\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:17.709453 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.709435    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769e37b8-624f-492c-9f56-dbd965280659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kj96s\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:17.711429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.711408    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769e37b8-624f-492c-9f56-dbd965280659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kj96s\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:17.721234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.721205    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbk8\" (UniqueName: \"kubernetes.io/projected/769e37b8-624f-492c-9f56-dbd965280659-kube-api-access-4cbk8\") pod \"maas-api-revoke-sub-85237e45-kj96s\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:17.918186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:17.918160    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:18.197916 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:18.197887    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"]
Nov 02 01:50:18.200239 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:50:18.200212    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769e37b8_624f_492c_9f56_dbd965280659.slice/crio-f2f10c2fff6284decda4fb66ca770a9e25680bf78ff95a3650077b38393b5092 WatchSource:0}: Error finding container f2f10c2fff6284decda4fb66ca770a9e25680bf78ff95a3650077b38393b5092: Status 404 returned error can't find the container with id f2f10c2fff6284decda4fb66ca770a9e25680bf78ff95a3650077b38393b5092
Nov 02 01:50:18.619897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:18.619857    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s" event={"ID":"769e37b8-624f-492c-9f56-dbd965280659","Type":"ContainerStarted","Data":"6ee5e08b4f468dec0fd11fbab2864ccebfd7fe8dd963324c3afcccaae39e4b94"}
Nov 02 01:50:18.620369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:18.619916    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s" event={"ID":"769e37b8-624f-492c-9f56-dbd965280659","Type":"ContainerStarted","Data":"f2f10c2fff6284decda4fb66ca770a9e25680bf78ff95a3650077b38393b5092"}
Nov 02 01:50:18.640231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:18.640173    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s" podStartSLOduration=1.640157215 podStartE2EDuration="1.640157215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:18.637554766 +0000 UTC m=+2123.942487592" watchObservedRunningTime="2026-11-02 01:50:18.640157215 +0000 UTC m=+2123.945090075"
Nov 02 01:50:19.626261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:19.626226    4851 generic.go:356] "Generic (PLEG): container finished" podID="769e37b8-624f-492c-9f56-dbd965280659" containerID="6ee5e08b4f468dec0fd11fbab2864ccebfd7fe8dd963324c3afcccaae39e4b94" exitCode=0
Nov 02 01:50:19.626680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:19.626318    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s" event={"ID":"769e37b8-624f-492c-9f56-dbd965280659","Type":"ContainerDied","Data":"6ee5e08b4f468dec0fd11fbab2864ccebfd7fe8dd963324c3afcccaae39e4b94"}
Nov 02 01:50:20.769106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:20.769079    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:20.837148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:20.837113    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/769e37b8-624f-492c-9f56-dbd965280659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769e37b8-624f-492c-9f56-dbd965280659-maas-api-xks-ca\") pod \"769e37b8-624f-492c-9f56-dbd965280659\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") "
Nov 02 01:50:20.837356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:20.837161    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769e37b8-624f-492c-9f56-dbd965280659-kube-api-access-4cbk8\" (UniqueName: \"kubernetes.io/projected/769e37b8-624f-492c-9f56-dbd965280659-kube-api-access-4cbk8\") pod \"769e37b8-624f-492c-9f56-dbd965280659\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") "
Nov 02 01:50:20.837356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:20.837194    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769e37b8-624f-492c-9f56-dbd965280659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769e37b8-624f-492c-9f56-dbd965280659-maas-api-openshift-ca\") pod \"769e37b8-624f-492c-9f56-dbd965280659\" (UID: \"769e37b8-624f-492c-9f56-dbd965280659\") "
Nov 02 01:50:20.837678 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:20.837646    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769e37b8-624f-492c-9f56-dbd965280659-maas-api-openshift-ca" pod "769e37b8-624f-492c-9f56-dbd965280659" (UID: "769e37b8-624f-492c-9f56-dbd965280659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:50:20.839368 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:20.839347    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/769e37b8-624f-492c-9f56-dbd965280659-maas-api-xks-ca" pod "769e37b8-624f-492c-9f56-dbd965280659" (UID: "769e37b8-624f-492c-9f56-dbd965280659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:50:20.839651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:20.839637    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769e37b8-624f-492c-9f56-dbd965280659-kube-api-access-4cbk8" pod "769e37b8-624f-492c-9f56-dbd965280659" (UID: "769e37b8-624f-492c-9f56-dbd965280659"). InnerVolumeSpecName "kube-api-access-4cbk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:50:20.938765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:20.938725    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cbk8\" (UniqueName: \"kubernetes.io/projected/769e37b8-624f-492c-9f56-dbd965280659-kube-api-access-4cbk8\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:50:20.938765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:20.938761    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769e37b8-624f-492c-9f56-dbd965280659-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:50:20.938765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:20.938771    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769e37b8-624f-492c-9f56-dbd965280659-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:50:21.638941 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:21.638906    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s" event={"ID":"769e37b8-624f-492c-9f56-dbd965280659","Type":"ContainerDied","Data":"f2f10c2fff6284decda4fb66ca770a9e25680bf78ff95a3650077b38393b5092"}
Nov 02 01:50:21.639133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:21.638960    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2f10c2fff6284decda4fb66ca770a9e25680bf78ff95a3650077b38393b5092"
Nov 02 01:50:21.639133 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:21.638915    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"
Nov 02 01:50:25.740485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.740440    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"]
Nov 02 01:50:25.740892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.740844    4851 cpu_manager.go:414] "containerMap: removing container" podUID="769e37b8-624f-492c-9f56-dbd965280659" containerName="revoke-keys"
Nov 02 01:50:25.740892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.740855    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769e37b8-624f-492c-9f56-dbd965280659" containerName="revoke-keys"
Nov 02 01:50:25.740968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.740937    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="769e37b8-624f-492c-9f56-dbd965280659" containerName="revoke-keys"
Nov 02 01:50:25.744196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.744181    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.758091 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.758049    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"]
Nov 02 01:50:25.761176 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.761148    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"]
Nov 02 01:50:25.764908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.764890    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.778192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.778161    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"]
Nov 02 01:50:25.785564 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.785540    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.785725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.785575    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.785725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.785609    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.785875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.785753    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.785875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.785796    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.785875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.785840    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.786009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.785883    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4srp\" (UniqueName: \"kubernetes.io/projected/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-kube-api-access-s4srp\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.786009 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.785976    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.786081 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.786037    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.886979 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.886936    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.887172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.886995    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.887172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887075    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.887172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887113    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.887172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887163    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.887403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887188    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5lsc\" (UniqueName: \"kubernetes.io/projected/2aa0e67e-2315-4bba-8060-e7d7b101918c-kube-api-access-k5lsc\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.887403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887210    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.887403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887231    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.887403 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887363    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.887682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887406    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.887682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887429    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.887682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887532    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.887682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887572    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.887682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887597    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4srp\" (UniqueName: \"kubernetes.io/projected/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-kube-api-access-s4srp\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.887682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887653    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.887682 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887683    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.888059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887732    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.888059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887759    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.888059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887784    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa0e67e-2315-4bba-8060-e7d7b101918c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.888059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887815    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.888059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887927    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.888059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.887956    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.888059 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.888014    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.889960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.889941    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.890121 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.890102    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.899655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.899626    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.899966 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.899944    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4srp\" (UniqueName: \"kubernetes.io/projected/cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c-kube-api-access-s4srp\") pod \"maas-default-gateway-openshift-default-76597d6855-kw5w7\" (UID: \"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:25.988898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.988862    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5lsc\" (UniqueName: \"kubernetes.io/projected/2aa0e67e-2315-4bba-8060-e7d7b101918c-kube-api-access-k5lsc\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.988898 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.988901    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.988919    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.988950    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.989000    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.989045    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.989069    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa0e67e-2315-4bba-8060-e7d7b101918c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.989103    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.989169    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.989387    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.989457    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989562 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.989524    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989802 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.989631    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.989956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.989934    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa0e67e-2315-4bba-8060-e7d7b101918c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.991585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.991542    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:25.991643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:25.991617    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:26.001417 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.001388    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5lsc\" (UniqueName: \"kubernetes.io/projected/2aa0e67e-2315-4bba-8060-e7d7b101918c-kube-api-access-k5lsc\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:26.001522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.001432    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa0e67e-2315-4bba-8060-e7d7b101918c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dvgpc\" (UID: \"2aa0e67e-2315-4bba-8060-e7d7b101918c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:26.058391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.058352    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:26.081995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.081969    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:26.344606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.344583    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"]
Nov 02 01:50:26.346574 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:50:26.346544    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdfb1d0_b327_48d5_8b80_f8ad7a991a1c.slice/crio-b435b80d8cbf04e7b35c2c55c651fc6ee2f0ba05775da60c6211b9c4a5886cf0 WatchSource:0}: Error finding container b435b80d8cbf04e7b35c2c55c651fc6ee2f0ba05775da60c6211b9c4a5886cf0: Status 404 returned error can't find the container with id b435b80d8cbf04e7b35c2c55c651fc6ee2f0ba05775da60c6211b9c4a5886cf0
Nov 02 01:50:26.348708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.348672    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:26.348834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.348749    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:26.348834 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.348778    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:26.374429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.374406    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"]
Nov 02 01:50:26.380606 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:50:26.380577    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa0e67e_2315_4bba_8060_e7d7b101918c.slice/crio-6be1ef30feb4c92a4043f5a5f8a6bc14445726e6892a54a5a66e4981d56df8ad WatchSource:0}: Error finding container 6be1ef30feb4c92a4043f5a5f8a6bc14445726e6892a54a5a66e4981d56df8ad: Status 404 returned error can't find the container with id 6be1ef30feb4c92a4043f5a5f8a6bc14445726e6892a54a5a66e4981d56df8ad
Nov 02 01:50:26.383365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.383330    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:26.383452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.383419    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:26.383489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.383461    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:26.673334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.672539    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" event={"ID":"2aa0e67e-2315-4bba-8060-e7d7b101918c","Type":"ContainerStarted","Data":"fda4054ceaffd08db04bdc12683c850bab756447da7cba0f349a9b2f3691fe09"}
Nov 02 01:50:26.673334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.672663    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" event={"ID":"2aa0e67e-2315-4bba-8060-e7d7b101918c","Type":"ContainerStarted","Data":"6be1ef30feb4c92a4043f5a5f8a6bc14445726e6892a54a5a66e4981d56df8ad"}
Nov 02 01:50:26.682524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.674910    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" event={"ID":"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c","Type":"ContainerStarted","Data":"608f2423bb37b60472871a2b2433ad661a92f1e448b9503db9d429186f5f1459"}
Nov 02 01:50:26.682524 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.674978    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" event={"ID":"cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c","Type":"ContainerStarted","Data":"b435b80d8cbf04e7b35c2c55c651fc6ee2f0ba05775da60c6211b9c4a5886cf0"}
Nov 02 01:50:26.701486 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.701422    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" podStartSLOduration=1.701403978 podStartE2EDuration="1.701403978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:26.697963791 +0000 UTC m=+2132.002896617" watchObservedRunningTime="2026-11-02 01:50:26.701403978 +0000 UTC m=+2132.006336804"
Nov 02 01:50:26.722961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:26.722904    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" podStartSLOduration=1.72288921 podStartE2EDuration="1.72288921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:26.720410895 +0000 UTC m=+2132.025343720" watchObservedRunningTime="2026-11-02 01:50:26.72288921 +0000 UTC m=+2132.027822036"
Nov 02 01:50:27.060968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:27.060471    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:27.082708 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:27.082657    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kw5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 02 01:50:27.082950 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:27.082729    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" podUID="cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 02 01:50:27.083377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:27.083353    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:27.084597 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:27.084575    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dvgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Nov 02 01:50:27.084695 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:27.084618    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" podUID="2aa0e67e-2315-4bba-8060-e7d7b101918c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Nov 02 01:50:28.060249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:28.059743    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kw5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 02 01:50:28.060249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:28.059815    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" podUID="cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 02 01:50:28.083746 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:28.083699    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dvgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Nov 02 01:50:28.084272 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:28.083769    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" podUID="2aa0e67e-2315-4bba-8060-e7d7b101918c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Nov 02 01:50:29.063232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:29.062771    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kw5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 02 01:50:29.063232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:29.062837    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" podUID="cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 02 01:50:29.083319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:29.082728    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dvgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Nov 02 01:50:29.083319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:29.082800    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" podUID="2aa0e67e-2315-4bba-8060-e7d7b101918c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Nov 02 01:50:30.063316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:30.062804    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kw5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 02 01:50:30.063316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:30.062869    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" podUID="cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 02 01:50:30.083023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:30.082980    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dvgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Nov 02 01:50:30.083284 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:30.083264    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" podUID="2aa0e67e-2315-4bba-8060-e7d7b101918c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Nov 02 01:50:31.061251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:31.060775    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kw5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 02 01:50:31.061251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:31.060841    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" podUID="cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 02 01:50:31.083473 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:31.083431    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dvgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Nov 02 01:50:31.084019 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:31.083498    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" podUID="2aa0e67e-2315-4bba-8060-e7d7b101918c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Nov 02 01:50:32.069317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:32.066760    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kw5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 02 01:50:32.069317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:32.066824    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" podUID="cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 02 01:50:32.084334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:32.083365    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dvgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Nov 02 01:50:32.084334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:32.083427    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" podUID="2aa0e67e-2315-4bba-8060-e7d7b101918c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Nov 02 01:50:33.059673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:33.059635    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kw5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 02 01:50:33.059891 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:33.059688    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" podUID="cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 02 01:50:33.083208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:33.083180    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dvgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Nov 02 01:50:33.083420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:33.083223    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" podUID="2aa0e67e-2315-4bba-8060-e7d7b101918c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Nov 02 01:50:34.659980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:34.659947    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:34.660786 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:34.660765    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:34.661788 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:34.661757    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc"
Nov 02 01:50:34.727564 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:34.727533    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:34.728342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:34.728314    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:34.729001 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:50:34.728978    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7"
Nov 02 01:50:47.654563 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:50:47.654524    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:51:03.155224 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:03.155189    4851 scope.go:122] "RemoveContainer" containerID="b52e8c1ceee9c50e4d4a3ace5b0e125e428de75f5ee18c230a9c31aa1f894e1c"
Nov 02 01:51:03.164796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:03.164775    4851 scope.go:122] "RemoveContainer" containerID="e41050565b293c3c2ccd180631c8011edcf24c362c2768fdc58c90937a1d4060"
Nov 02 01:51:03.173234 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:03.173216    4851 scope.go:122] "RemoveContainer" containerID="3176538d307b519b10a82ae5be1b7610202baca8b4329f998c70e56a32af20bd"
Nov 02 01:51:03.181111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:03.181095    4851 scope.go:122] "RemoveContainer" containerID="a8762c622475e758b92e0a854472d0eb8b9267717357ce0cfe2ecfd30a17bdee"
Nov 02 01:51:03.189139 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:03.189115    4851 scope.go:122] "RemoveContainer" containerID="d0a6286ce85e8e1bcfc6d16bb8a80661299f8f5bb1f55f7b3fe1d0ee40402d44"
Nov 02 01:51:03.197235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:03.197210    4851 scope.go:122] "RemoveContainer" containerID="686e7f34449c4cfaee7367d23925e7bc99c59c8165fc718fefaef9dce7cf642e"
Nov 02 01:51:11.402664 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.402627    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"]
Nov 02 01:51:11.408860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.408830    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.412913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.412883    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:51:11.413153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.413128    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"]
Nov 02 01:51:11.413865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.413840    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:51:11.413865 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.413859    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:51:11.414058 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.414004    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:51:11.494899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.494869    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pplt4\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.494899 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.494900    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbrt\" (UniqueName: \"kubernetes.io/projected/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-kube-api-access-4dbrt\") pod \"maas-api-revoke-sub-c1e271a9-pplt4\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.495136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.495029    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pplt4\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.595974 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.595936    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pplt4\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.596173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.596014    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pplt4\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.596173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.596032    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbrt\" (UniqueName: \"kubernetes.io/projected/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-kube-api-access-4dbrt\") pod \"maas-api-revoke-sub-c1e271a9-pplt4\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.596610 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.596586    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pplt4\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.598680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.598653    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pplt4\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.608942 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.608915    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbrt\" (UniqueName: \"kubernetes.io/projected/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-kube-api-access-4dbrt\") pod \"maas-api-revoke-sub-c1e271a9-pplt4\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.724226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.724192    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:11.999550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:11.999524    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"]
Nov 02 01:51:12.001877 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:51:12.001848    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3c6e6e_61a3_453d_aae4_72f9da7e2144.slice/crio-d3e75df6e267602dc3079174651ae3abb4655d182373348cf5e70b9f22e2a19c WatchSource:0}: Error finding container d3e75df6e267602dc3079174651ae3abb4655d182373348cf5e70b9f22e2a19c: Status 404 returned error can't find the container with id d3e75df6e267602dc3079174651ae3abb4655d182373348cf5e70b9f22e2a19c
Nov 02 01:51:12.980831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:12.980792    4851 generic.go:356] "Generic (PLEG): container finished" podID="9e3c6e6e-61a3-453d-aae4-72f9da7e2144" containerID="6739e8b10308f022ab189eb5c34fc285a5ab89971386e27a133c2022d86d0674" exitCode=0
Nov 02 01:51:12.980831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:12.980837    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4" event={"ID":"9e3c6e6e-61a3-453d-aae4-72f9da7e2144","Type":"ContainerDied","Data":"6739e8b10308f022ab189eb5c34fc285a5ab89971386e27a133c2022d86d0674"}
Nov 02 01:51:12.981336 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:12.980874    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4" event={"ID":"9e3c6e6e-61a3-453d-aae4-72f9da7e2144","Type":"ContainerStarted","Data":"d3e75df6e267602dc3079174651ae3abb4655d182373348cf5e70b9f22e2a19c"}
Nov 02 01:51:14.121431 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.121408    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:14.217393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.217340    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-openshift-ca\") pod \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") "
Nov 02 01:51:14.217393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.217407    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-kube-api-access-4dbrt\" (UniqueName: \"kubernetes.io/projected/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-kube-api-access-4dbrt\") pod \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") "
Nov 02 01:51:14.217659 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.217510    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-xks-ca\") pod \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\" (UID: \"9e3c6e6e-61a3-453d-aae4-72f9da7e2144\") "
Nov 02 01:51:14.217781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.217758    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-openshift-ca" pod "9e3c6e6e-61a3-453d-aae4-72f9da7e2144" (UID: "9e3c6e6e-61a3-453d-aae4-72f9da7e2144"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:14.219651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.219629    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-kube-api-access-4dbrt" pod "9e3c6e6e-61a3-453d-aae4-72f9da7e2144" (UID: "9e3c6e6e-61a3-453d-aae4-72f9da7e2144"). InnerVolumeSpecName "kube-api-access-4dbrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:14.219731 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.219708    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-xks-ca" pod "9e3c6e6e-61a3-453d-aae4-72f9da7e2144" (UID: "9e3c6e6e-61a3-453d-aae4-72f9da7e2144"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:51:14.318609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.318537    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:51:14.318609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.318567    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:51:14.318609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.318578    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dbrt\" (UniqueName: \"kubernetes.io/projected/9e3c6e6e-61a3-453d-aae4-72f9da7e2144-kube-api-access-4dbrt\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:51:14.993233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.993204    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"
Nov 02 01:51:14.993436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.993232    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4" event={"ID":"9e3c6e6e-61a3-453d-aae4-72f9da7e2144","Type":"ContainerDied","Data":"d3e75df6e267602dc3079174651ae3abb4655d182373348cf5e70b9f22e2a19c"}
Nov 02 01:51:14.993436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:14.993279    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3e75df6e267602dc3079174651ae3abb4655d182373348cf5e70b9f22e2a19c"
Nov 02 01:51:17.654111 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:51:17.654078    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:51:30.137530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.137487    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"]
Nov 02 01:51:30.138067 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.138047    4851 cpu_manager.go:414] "containerMap: removing container" podUID="9e3c6e6e-61a3-453d-aae4-72f9da7e2144" containerName="revoke-keys"
Nov 02 01:51:30.138158 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.138071    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3c6e6e-61a3-453d-aae4-72f9da7e2144" containerName="revoke-keys"
Nov 02 01:51:30.138242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.138229    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3c6e6e-61a3-453d-aae4-72f9da7e2144" containerName="revoke-keys"
Nov 02 01:51:30.141305 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.141269    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.149432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.148559    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:51:30.149432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.148992    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:51:30.149432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.149241    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:51:30.149432 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.148656    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:51:30.152148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.152125    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"]
Nov 02 01:51:30.212944 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.212906    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ltzgv\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.213141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.212981    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ltzgv\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.213141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.213072    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69w5\" (UniqueName: \"kubernetes.io/projected/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-kube-api-access-k69w5\") pod \"maas-api-revoke-sub-2e1f5b5f-ltzgv\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.314337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.314274    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ltzgv\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.314537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.314358    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69w5\" (UniqueName: \"kubernetes.io/projected/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-kube-api-access-k69w5\") pod \"maas-api-revoke-sub-2e1f5b5f-ltzgv\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.314537 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.314452    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ltzgv\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.315096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.315068    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ltzgv\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.317035 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.317007    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-ltzgv\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.327526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.327506    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69w5\" (UniqueName: \"kubernetes.io/projected/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-kube-api-access-k69w5\") pod \"maas-api-revoke-sub-2e1f5b5f-ltzgv\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.457716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.457673    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:30.734913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:30.734884    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"]
Nov 02 01:51:30.736811 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:51:30.736788    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f15e4a3_9a5b_40a6_b929_9b2e9ed00581.slice/crio-43c9701512b0ccd0b4e03f19081895e5c060d02b8161bd403bd90a6628da8407 WatchSource:0}: Error finding container 43c9701512b0ccd0b4e03f19081895e5c060d02b8161bd403bd90a6628da8407: Status 404 returned error can't find the container with id 43c9701512b0ccd0b4e03f19081895e5c060d02b8161bd403bd90a6628da8407
Nov 02 01:51:31.133092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:31.133054    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv" event={"ID":"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581","Type":"ContainerStarted","Data":"e3ab35d647cb081accc81e0dbd0d87715a002f01218f12aceb363e312d91214a"}
Nov 02 01:51:31.133261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:31.133120    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv" event={"ID":"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581","Type":"ContainerStarted","Data":"43c9701512b0ccd0b4e03f19081895e5c060d02b8161bd403bd90a6628da8407"}
Nov 02 01:51:31.153169 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:31.153120    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv" podStartSLOduration=1.153101964 podStartE2EDuration="1.153101964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:31.151320779 +0000 UTC m=+2196.456253606" watchObservedRunningTime="2026-11-02 01:51:31.153101964 +0000 UTC m=+2196.458035191"
Nov 02 01:51:32.140781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:32.140747    4851 generic.go:356] "Generic (PLEG): container finished" podID="2f15e4a3-9a5b-40a6-b929-9b2e9ed00581" containerID="e3ab35d647cb081accc81e0dbd0d87715a002f01218f12aceb363e312d91214a" exitCode=0
Nov 02 01:51:32.140986 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:32.140845    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv" event={"ID":"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581","Type":"ContainerDied","Data":"e3ab35d647cb081accc81e0dbd0d87715a002f01218f12aceb363e312d91214a"}
Nov 02 01:51:33.025845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.025802    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"]
Nov 02 01:51:33.028176 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.028149    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57db3b80-s5fdk"]
Nov 02 01:51:33.282026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.281972    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:33.444228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.444194    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-openshift-ca\") pod \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") "
Nov 02 01:51:33.444450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.444308    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-xks-ca\") pod \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") "
Nov 02 01:51:33.444450 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.444402    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-kube-api-access-k69w5\" (UniqueName: \"kubernetes.io/projected/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-kube-api-access-k69w5\") pod \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\" (UID: \"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581\") "
Nov 02 01:51:33.444608 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.444583    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-openshift-ca" pod "2f15e4a3-9a5b-40a6-b929-9b2e9ed00581" (UID: "2f15e4a3-9a5b-40a6-b929-9b2e9ed00581"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:33.444732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.444716    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:51:33.446570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.446540    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-kube-api-access-k69w5" pod "2f15e4a3-9a5b-40a6-b929-9b2e9ed00581" (UID: "2f15e4a3-9a5b-40a6-b929-9b2e9ed00581"). InnerVolumeSpecName "kube-api-access-k69w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:33.446681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.446623    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-xks-ca" pod "2f15e4a3-9a5b-40a6-b929-9b2e9ed00581" (UID: "2f15e4a3-9a5b-40a6-b929-9b2e9ed00581"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:51:33.545510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.545421    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:51:33.545510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:33.545458    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69w5\" (UniqueName: \"kubernetes.io/projected/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581-kube-api-access-k69w5\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:51:34.155722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:34.155692    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"
Nov 02 01:51:34.156247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:34.155737    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv" event={"ID":"2f15e4a3-9a5b-40a6-b929-9b2e9ed00581","Type":"ContainerDied","Data":"43c9701512b0ccd0b4e03f19081895e5c060d02b8161bd403bd90a6628da8407"}
Nov 02 01:51:34.156247 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:34.155796    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43c9701512b0ccd0b4e03f19081895e5c060d02b8161bd403bd90a6628da8407"
Nov 02 01:51:34.844546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:34.844512    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a" path="/var/lib/kubelet/pods/1b58949e-ba8f-4fd8-8d00-b6b1faeb6b7a/volumes"
Nov 02 01:51:35.028673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:35.028586    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"]
Nov 02 01:51:35.030621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:35.030598    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bb21da-lldvk"]
Nov 02 01:51:36.845214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:36.845179    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2" path="/var/lib/kubelet/pods/f4e2ac7b-34ce-4d14-bf77-479ebc1f35f2/volumes"
Nov 02 01:51:41.231874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.231837    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"]
Nov 02 01:51:41.232373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.232225    4851 cpu_manager.go:414] "containerMap: removing container" podUID="2f15e4a3-9a5b-40a6-b929-9b2e9ed00581" containerName="revoke-keys"
Nov 02 01:51:41.232373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.232235    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f15e4a3-9a5b-40a6-b929-9b2e9ed00581" containerName="revoke-keys"
Nov 02 01:51:41.232373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.232341    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f15e4a3-9a5b-40a6-b929-9b2e9ed00581" containerName="revoke-keys"
Nov 02 01:51:41.236644 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.236617    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.242617 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.242593    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:51:41.242813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.242620    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:51:41.242927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.242656    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:51:41.242927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.242737    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:51:41.245971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.245478    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"]
Nov 02 01:51:41.416545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.416507    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-42pfv\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.416764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.416557    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv5p4\" (UniqueName: \"kubernetes.io/projected/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-kube-api-access-gv5p4\") pod \"maas-api-revoke-sub-ec843018-42pfv\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.416764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.416697    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-42pfv\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.517639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.517546    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-42pfv\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.517844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.517677    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-42pfv\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.517844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.517709    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv5p4\" (UniqueName: \"kubernetes.io/projected/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-kube-api-access-gv5p4\") pod \"maas-api-revoke-sub-ec843018-42pfv\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.518498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.518472    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-42pfv\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.520356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.520329    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-42pfv\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.530502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.530470    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv5p4\" (UniqueName: \"kubernetes.io/projected/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-kube-api-access-gv5p4\") pod \"maas-api-revoke-sub-ec843018-42pfv\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.553175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.553146    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:41.828536 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:41.828510    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"]
Nov 02 01:51:41.831329 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:51:41.831282    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f77fb0_bd5d_4e20_a0fc_bf5adb424c9b.slice/crio-79e7cea922269039d2e95a676528ab4e333e78eaf4e364f9a471c0957c0372e5 WatchSource:0}: Error finding container 79e7cea922269039d2e95a676528ab4e333e78eaf4e364f9a471c0957c0372e5: Status 404 returned error can't find the container with id 79e7cea922269039d2e95a676528ab4e333e78eaf4e364f9a471c0957c0372e5
Nov 02 01:51:42.202584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:42.202545    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv" event={"ID":"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b","Type":"ContainerStarted","Data":"3608bc4f9fcfe78c44182551c8066ee58c94e5e7e25d0eb11b73879c478dac9b"}
Nov 02 01:51:42.202927 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:42.202601    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv" event={"ID":"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b","Type":"ContainerStarted","Data":"79e7cea922269039d2e95a676528ab4e333e78eaf4e364f9a471c0957c0372e5"}
Nov 02 01:51:42.222235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:42.222184    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv" podStartSLOduration=1.222169217 podStartE2EDuration="1.222169217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:42.221425474 +0000 UTC m=+2207.526358302" watchObservedRunningTime="2026-11-02 01:51:42.222169217 +0000 UTC m=+2207.527102043"
Nov 02 01:51:43.211464 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:43.211383    4851 generic.go:356] "Generic (PLEG): container finished" podID="b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b" containerID="3608bc4f9fcfe78c44182551c8066ee58c94e5e7e25d0eb11b73879c478dac9b" exitCode=0
Nov 02 01:51:43.212034 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:43.211551    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv" event={"ID":"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b","Type":"ContainerDied","Data":"3608bc4f9fcfe78c44182551c8066ee58c94e5e7e25d0eb11b73879c478dac9b"}
Nov 02 01:51:45.032312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.031489    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:45.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.198465    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-openshift-ca\") pod \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") "
Nov 02 01:51:45.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.198536    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-kube-api-access-gv5p4\" (UniqueName: \"kubernetes.io/projected/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-kube-api-access-gv5p4\") pod \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") "
Nov 02 01:51:45.205316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.198684    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-xks-ca\") pod \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\" (UID: \"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b\") "
Nov 02 01:51:45.230316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.226850    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-openshift-ca" pod "b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b" (UID: "b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:45.230316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.227389    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:51:45.320673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.320346    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv" event={"ID":"b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b","Type":"ContainerDied","Data":"79e7cea922269039d2e95a676528ab4e333e78eaf4e364f9a471c0957c0372e5"}
Nov 02 01:51:45.320673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.320409    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79e7cea922269039d2e95a676528ab4e333e78eaf4e364f9a471c0957c0372e5"
Nov 02 01:51:45.320673 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.320489    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"
Nov 02 01:51:45.346090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.346011    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-kube-api-access-gv5p4" pod "b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b" (UID: "b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b"). InnerVolumeSpecName "kube-api-access-gv5p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:45.346872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.346832    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-xks-ca" pod "b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b" (UID: "b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:51:45.442984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.442901    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:51:45.442984 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:51:45.442950    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv5p4\" (UniqueName: \"kubernetes.io/projected/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b-kube-api-access-gv5p4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:51:47.655336 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:51:47.655277    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:52:03.241273 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:03.241242    4851 scope.go:122] "RemoveContainer" containerID="4d5413c76706ea2197b5a4d911547d2cf90683e6ae8d26158802f2ecdcac3cc2"
Nov 02 01:52:03.253242 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:03.253223    4851 scope.go:122] "RemoveContainer" containerID="7e1dfe401d59f064cd51e09d2b12f33e3f089e92f72d9b8cf4b56c63a15c2c78"
Nov 02 01:52:03.261607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:03.261586    4851 scope.go:122] "RemoveContainer" containerID="0664d0d4e62bbe375396360b4c51b584674e5325f1a694a2460a200ec64ff0fd"
Nov 02 01:52:03.269741 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:03.269722    4851 scope.go:122] "RemoveContainer" containerID="7722b6954b95b1264b1dca275ae21eeb885a38f6f1439e69c1b2cc3cd8cb798b"
Nov 02 01:52:03.277726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:03.277710    4851 scope.go:122] "RemoveContainer" containerID="0786562154e5e9ad1d263c4313b12d1725ccfcd07f4c346f419b59f75bdc0fc9"
Nov 02 01:52:03.285861 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:03.285834    4851 scope.go:122] "RemoveContainer" containerID="a10124dcd90bec9a41e741129d9117ea2b3f7c252b8f2a427bc9de3314c74edf"
Nov 02 01:52:07.868232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.868194    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"]
Nov 02 01:52:07.868633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.868600    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b" containerName="revoke-keys"
Nov 02 01:52:07.868633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.868611    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b" containerName="revoke-keys"
Nov 02 01:52:07.868724 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.868701    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b" containerName="revoke-keys"
Nov 02 01:52:07.873905 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.873880    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:07.880598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.878883    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:52:07.880598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.879431    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:52:07.880598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.879520    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:52:07.880598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.879598    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:52:07.881182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.881155    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"]
Nov 02 01:52:07.940327 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.940269    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7lg\" (UniqueName: \"kubernetes.io/projected/88bc44b1-175a-4f62-8753-1b5ac9613bfc-kube-api-access-7c7lg\") pod \"maas-api-revoke-sub-046e1343-kv2vm\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:07.940535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.940471    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kv2vm\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:07.940596 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:07.940549    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kv2vm\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:08.041468 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:08.041440    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kv2vm\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:08.041660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:08.041504    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7lg\" (UniqueName: \"kubernetes.io/projected/88bc44b1-175a-4f62-8753-1b5ac9613bfc-kube-api-access-7c7lg\") pod \"maas-api-revoke-sub-046e1343-kv2vm\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:08.041660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:08.041548    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kv2vm\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:08.042088 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:08.042057    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kv2vm\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:08.044039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:08.044017    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kv2vm\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:08.054198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:08.054168    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7lg\" (UniqueName: \"kubernetes.io/projected/88bc44b1-175a-4f62-8753-1b5ac9613bfc-kube-api-access-7c7lg\") pod \"maas-api-revoke-sub-046e1343-kv2vm\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:08.192205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:08.192169    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:08.469208 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:08.469125    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"]
Nov 02 01:52:08.472281 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:52:08.472253    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bc44b1_175a_4f62_8753_1b5ac9613bfc.slice/crio-91aba975347d4ecf6d2aebab518187e6ffde1e64328f06f2a65623b6cd0c7196 WatchSource:0}: Error finding container 91aba975347d4ecf6d2aebab518187e6ffde1e64328f06f2a65623b6cd0c7196: Status 404 returned error can't find the container with id 91aba975347d4ecf6d2aebab518187e6ffde1e64328f06f2a65623b6cd0c7196
Nov 02 01:52:08.508787 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:08.508566    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm" event={"ID":"88bc44b1-175a-4f62-8753-1b5ac9613bfc","Type":"ContainerStarted","Data":"91aba975347d4ecf6d2aebab518187e6ffde1e64328f06f2a65623b6cd0c7196"}
Nov 02 01:52:09.515728 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:09.515640    4851 generic.go:356] "Generic (PLEG): container finished" podID="88bc44b1-175a-4f62-8753-1b5ac9613bfc" containerID="52d0bf54ce3211ee224444220228f4676d4997638b695cb7d23bb87eb544b314" exitCode=0
Nov 02 01:52:09.516095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:09.515722    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm" event={"ID":"88bc44b1-175a-4f62-8753-1b5ac9613bfc","Type":"ContainerDied","Data":"52d0bf54ce3211ee224444220228f4676d4997638b695cb7d23bb87eb544b314"}
Nov 02 01:52:10.662765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:10.662742    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:10.765710 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:10.765673    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-xks-ca\") pod \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") "
Nov 02 01:52:10.765922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:10.765724    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88bc44b1-175a-4f62-8753-1b5ac9613bfc-kube-api-access-7c7lg\" (UniqueName: \"kubernetes.io/projected/88bc44b1-175a-4f62-8753-1b5ac9613bfc-kube-api-access-7c7lg\") pod \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") "
Nov 02 01:52:10.765922 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:10.765766    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-openshift-ca\") pod \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\" (UID: \"88bc44b1-175a-4f62-8753-1b5ac9613bfc\") "
Nov 02 01:52:10.766230 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:10.766206    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-openshift-ca" pod "88bc44b1-175a-4f62-8753-1b5ac9613bfc" (UID: "88bc44b1-175a-4f62-8753-1b5ac9613bfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:52:10.767937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:10.767918    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-xks-ca" pod "88bc44b1-175a-4f62-8753-1b5ac9613bfc" (UID: "88bc44b1-175a-4f62-8753-1b5ac9613bfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:52:10.768057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:10.768042    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bc44b1-175a-4f62-8753-1b5ac9613bfc-kube-api-access-7c7lg" pod "88bc44b1-175a-4f62-8753-1b5ac9613bfc" (UID: "88bc44b1-175a-4f62-8753-1b5ac9613bfc"). InnerVolumeSpecName "kube-api-access-7c7lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:10.866939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:10.866907    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:52:10.866939 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:10.866935    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c7lg\" (UniqueName: \"kubernetes.io/projected/88bc44b1-175a-4f62-8753-1b5ac9613bfc-kube-api-access-7c7lg\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:52:10.867128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:10.866946    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bc44b1-175a-4f62-8753-1b5ac9613bfc-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:52:11.527486 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:11.527455    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"
Nov 02 01:52:11.527704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:11.527497    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm" event={"ID":"88bc44b1-175a-4f62-8753-1b5ac9613bfc","Type":"ContainerDied","Data":"91aba975347d4ecf6d2aebab518187e6ffde1e64328f06f2a65623b6cd0c7196"}
Nov 02 01:52:11.527704 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:11.527542    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91aba975347d4ecf6d2aebab518187e6ffde1e64328f06f2a65623b6cd0c7196"
Nov 02 01:52:17.655475 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:52:17.655436    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:52:18.085495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.085458    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"]
Nov 02 01:52:18.085889 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.085877    4851 cpu_manager.go:414] "containerMap: removing container" podUID="88bc44b1-175a-4f62-8753-1b5ac9613bfc" containerName="revoke-keys"
Nov 02 01:52:18.085937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.085892    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88bc44b1-175a-4f62-8753-1b5ac9613bfc" containerName="revoke-keys"
Nov 02 01:52:18.085977 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.085950    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="88bc44b1-175a-4f62-8753-1b5ac9613bfc" containerName="revoke-keys"
Nov 02 01:52:18.089362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.089338    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.095934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.095847    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:52:18.096144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.096121    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:52:18.096236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.096218    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:52:18.099699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.099670    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"]
Nov 02 01:52:18.100492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.100464    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:52:18.133978 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.133947    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-cs9hj\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.134154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.133987    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-cs9hj\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.134154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.134075    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbplj\" (UniqueName: \"kubernetes.io/projected/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-kube-api-access-fbplj\") pod \"maas-api-revoke-sub-d3fba056-cs9hj\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.234793 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.234762    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-cs9hj\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.234954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.234798    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-cs9hj\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.234954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.234835    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbplj\" (UniqueName: \"kubernetes.io/projected/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-kube-api-access-fbplj\") pod \"maas-api-revoke-sub-d3fba056-cs9hj\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.235541 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.235517    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-cs9hj\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.237307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.237271    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-cs9hj\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.247015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.246993    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbplj\" (UniqueName: \"kubernetes.io/projected/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-kube-api-access-fbplj\") pod \"maas-api-revoke-sub-d3fba056-cs9hj\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.407205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.407138    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:18.691970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:18.691944    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"]
Nov 02 01:52:18.694012 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:52:18.693982    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf241d58_ffd7_44e1_af14_f30aed6c7c0c.slice/crio-f66a45ba40eff44f734829b33c0981071e7987f41f17d882d54f592a62fb3aae WatchSource:0}: Error finding container f66a45ba40eff44f734829b33c0981071e7987f41f17d882d54f592a62fb3aae: Status 404 returned error can't find the container with id f66a45ba40eff44f734829b33c0981071e7987f41f17d882d54f592a62fb3aae
Nov 02 01:52:19.575356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:19.575158    4851 generic.go:356] "Generic (PLEG): container finished" podID="bf241d58-ffd7-44e1-af14-f30aed6c7c0c" containerID="6360fba19ff58281191520bf39e571a8baa352c98e3774d4b5502f57ff02077c" exitCode=0
Nov 02 01:52:19.575356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:19.575245    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj" event={"ID":"bf241d58-ffd7-44e1-af14-f30aed6c7c0c","Type":"ContainerDied","Data":"6360fba19ff58281191520bf39e571a8baa352c98e3774d4b5502f57ff02077c"}
Nov 02 01:52:19.575356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:19.575329    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj" event={"ID":"bf241d58-ffd7-44e1-af14-f30aed6c7c0c","Type":"ContainerStarted","Data":"f66a45ba40eff44f734829b33c0981071e7987f41f17d882d54f592a62fb3aae"}
Nov 02 01:52:21.303511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.303478    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:21.383315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.382653    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-xks-ca\") pod \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") "
Nov 02 01:52:21.383315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.382847    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-kube-api-access-fbplj\" (UniqueName: \"kubernetes.io/projected/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-kube-api-access-fbplj\") pod \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") "
Nov 02 01:52:21.383315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.382927    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-openshift-ca\") pod \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\" (UID: \"bf241d58-ffd7-44e1-af14-f30aed6c7c0c\") "
Nov 02 01:52:21.387314 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.383712    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-openshift-ca" pod "bf241d58-ffd7-44e1-af14-f30aed6c7c0c" (UID: "bf241d58-ffd7-44e1-af14-f30aed6c7c0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:52:21.462538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.462502    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-kube-api-access-fbplj" pod "bf241d58-ffd7-44e1-af14-f30aed6c7c0c" (UID: "bf241d58-ffd7-44e1-af14-f30aed6c7c0c"). InnerVolumeSpecName "kube-api-access-fbplj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:21.482515 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.482433    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-xks-ca" pod "bf241d58-ffd7-44e1-af14-f30aed6c7c0c" (UID: "bf241d58-ffd7-44e1-af14-f30aed6c7c0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:52:21.488321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.484879    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbplj\" (UniqueName: \"kubernetes.io/projected/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-kube-api-access-fbplj\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:52:21.488321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.484911    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:52:21.488321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.484930    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf241d58-ffd7-44e1-af14-f30aed6c7c0c-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:52:21.641326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.633539    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj" event={"ID":"bf241d58-ffd7-44e1-af14-f30aed6c7c0c","Type":"ContainerDied","Data":"f66a45ba40eff44f734829b33c0981071e7987f41f17d882d54f592a62fb3aae"}
Nov 02 01:52:21.641326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.640368    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f66a45ba40eff44f734829b33c0981071e7987f41f17d882d54f592a62fb3aae"
Nov 02 01:52:21.641326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:21.641133    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"
Nov 02 01:52:33.563316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:33.561142    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:52:33.563316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:33.561217    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:52:33.641318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:33.639979    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-ns4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:52:33.641318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:33.640038    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-ns4lc" podUID="0de30efc-e5bc-428c-ba8d-6b7967e29473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:52:38.749735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.749699    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"]
Nov 02 01:52:38.750174 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.750157    4851 cpu_manager.go:414] "containerMap: removing container" podUID="bf241d58-ffd7-44e1-af14-f30aed6c7c0c" containerName="revoke-keys"
Nov 02 01:52:38.750221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.750174    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf241d58-ffd7-44e1-af14-f30aed6c7c0c" containerName="revoke-keys"
Nov 02 01:52:38.750284 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.750271    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf241d58-ffd7-44e1-af14-f30aed6c7c0c" containerName="revoke-keys"
Nov 02 01:52:38.753758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.753737    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:38.760440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.760413    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:52:38.760607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.760492    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:52:38.760607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.760571    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:52:38.763532 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.763511    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:52:38.764399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.764373    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"]
Nov 02 01:52:38.794768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.794731    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-h96j9\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:38.794941 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.794854    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-h96j9\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:38.795015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.794938    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7qh\" (UniqueName: \"kubernetes.io/projected/8ad9143a-341f-4bc6-aecb-356f948ed422-kube-api-access-vw7qh\") pod \"maas-api-revoke-sub-ae594979-h96j9\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:38.895548 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.895508    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-h96j9\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:38.895751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.895562    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7qh\" (UniqueName: \"kubernetes.io/projected/8ad9143a-341f-4bc6-aecb-356f948ed422-kube-api-access-vw7qh\") pod \"maas-api-revoke-sub-ae594979-h96j9\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:38.895751 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.895619    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-h96j9\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:38.896161 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.896143    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-h96j9\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:38.898087 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.898065    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-h96j9\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:38.908069 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:38.908040    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7qh\" (UniqueName: \"kubernetes.io/projected/8ad9143a-341f-4bc6-aecb-356f948ed422-kube-api-access-vw7qh\") pod \"maas-api-revoke-sub-ae594979-h96j9\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:39.070498 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:39.070405    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:39.347806 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:39.347778    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"]
Nov 02 01:52:39.350038 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:52:39.350011    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad9143a_341f_4bc6_aecb_356f948ed422.slice/crio-0319bfff09e2b7e8c8ee44abf603fbee212e077570daea84aa0cf803be8e7bec WatchSource:0}: Error finding container 0319bfff09e2b7e8c8ee44abf603fbee212e077570daea84aa0cf803be8e7bec: Status 404 returned error can't find the container with id 0319bfff09e2b7e8c8ee44abf603fbee212e077570daea84aa0cf803be8e7bec
Nov 02 01:52:39.750990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:39.750945    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9" event={"ID":"8ad9143a-341f-4bc6-aecb-356f948ed422","Type":"ContainerStarted","Data":"16787300ffb52e8a1742abe508c2ff0abc667ef459fbb666a53fb4dd789693d9"}
Nov 02 01:52:39.751361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:39.751005    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9" event={"ID":"8ad9143a-341f-4bc6-aecb-356f948ed422","Type":"ContainerStarted","Data":"0319bfff09e2b7e8c8ee44abf603fbee212e077570daea84aa0cf803be8e7bec"}
Nov 02 01:52:39.771252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:39.771208    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9" podStartSLOduration=1.7711949439999999 podStartE2EDuration="1.771194944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:39.769273584 +0000 UTC m=+2265.074206411" watchObservedRunningTime="2026-11-02 01:52:39.771194944 +0000 UTC m=+2265.076127770"
Nov 02 01:52:40.758013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:40.757977    4851 generic.go:356] "Generic (PLEG): container finished" podID="8ad9143a-341f-4bc6-aecb-356f948ed422" containerID="16787300ffb52e8a1742abe508c2ff0abc667ef459fbb666a53fb4dd789693d9" exitCode=0
Nov 02 01:52:40.758462 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:40.758061    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9" event={"ID":"8ad9143a-341f-4bc6-aecb-356f948ed422","Type":"ContainerDied","Data":"16787300ffb52e8a1742abe508c2ff0abc667ef459fbb666a53fb4dd789693d9"}
Nov 02 01:52:41.908755 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:41.908729    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:42.027726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.027636    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-openshift-ca\") pod \"8ad9143a-341f-4bc6-aecb-356f948ed422\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") "
Nov 02 01:52:42.027726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.027707    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad9143a-341f-4bc6-aecb-356f948ed422-kube-api-access-vw7qh\" (UniqueName: \"kubernetes.io/projected/8ad9143a-341f-4bc6-aecb-356f948ed422-kube-api-access-vw7qh\") pod \"8ad9143a-341f-4bc6-aecb-356f948ed422\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") "
Nov 02 01:52:42.027947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.027745    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-xks-ca\") pod \"8ad9143a-341f-4bc6-aecb-356f948ed422\" (UID: \"8ad9143a-341f-4bc6-aecb-356f948ed422\") "
Nov 02 01:52:42.028136 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.028114    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-openshift-ca" pod "8ad9143a-341f-4bc6-aecb-356f948ed422" (UID: "8ad9143a-341f-4bc6-aecb-356f948ed422"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:52:42.030094 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.030067    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-xks-ca" pod "8ad9143a-341f-4bc6-aecb-356f948ed422" (UID: "8ad9143a-341f-4bc6-aecb-356f948ed422"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:52:42.030216 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.030137    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad9143a-341f-4bc6-aecb-356f948ed422-kube-api-access-vw7qh" pod "8ad9143a-341f-4bc6-aecb-356f948ed422" (UID: "8ad9143a-341f-4bc6-aecb-356f948ed422"). InnerVolumeSpecName "kube-api-access-vw7qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:42.129517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.129482    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:52:42.129517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.129513    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7qh\" (UniqueName: \"kubernetes.io/projected/8ad9143a-341f-4bc6-aecb-356f948ed422-kube-api-access-vw7qh\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:52:42.129517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.129523    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad9143a-341f-4bc6-aecb-356f948ed422-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:52:42.769667 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.769637    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"
Nov 02 01:52:42.769875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.769680    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9" event={"ID":"8ad9143a-341f-4bc6-aecb-356f948ed422","Type":"ContainerDied","Data":"0319bfff09e2b7e8c8ee44abf603fbee212e077570daea84aa0cf803be8e7bec"}
Nov 02 01:52:42.769875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:42.769760    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0319bfff09e2b7e8c8ee44abf603fbee212e077570daea84aa0cf803be8e7bec"
Nov 02 01:52:47.654852 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:52:47.654820    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:52:56.898670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.898636    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"]
Nov 02 01:52:56.899095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.899036    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8ad9143a-341f-4bc6-aecb-356f948ed422" containerName="revoke-keys"
Nov 02 01:52:56.899095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.899048    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad9143a-341f-4bc6-aecb-356f948ed422" containerName="revoke-keys"
Nov 02 01:52:56.899173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.899119    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad9143a-341f-4bc6-aecb-356f948ed422" containerName="revoke-keys"
Nov 02 01:52:56.902321 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.902281    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:56.908549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.908517    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:52:56.909857 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.909829    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:52:56.910756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.909891    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:52:56.910756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.909937    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:52:56.912894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.912872    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"]
Nov 02 01:52:56.987421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.987378    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tq85z\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:56.987421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.987425    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vw6\" (UniqueName: \"kubernetes.io/projected/4f90563c-5030-495b-83b5-de17d38a075e-kube-api-access-r4vw6\") pod \"maas-api-revoke-sub-a4aef7d7-tq85z\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:56.987640 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:56.987529    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tq85z\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:57.088259 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.088216    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tq85z\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:57.088500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.088271    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vw6\" (UniqueName: \"kubernetes.io/projected/4f90563c-5030-495b-83b5-de17d38a075e-kube-api-access-r4vw6\") pod \"maas-api-revoke-sub-a4aef7d7-tq85z\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:57.088500 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.088362    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tq85z\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:57.089005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.088985    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tq85z\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:57.090844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.090822    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tq85z\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:57.101226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.101192    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vw6\" (UniqueName: \"kubernetes.io/projected/4f90563c-5030-495b-83b5-de17d38a075e-kube-api-access-r4vw6\") pod \"maas-api-revoke-sub-a4aef7d7-tq85z\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:57.216516 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.216477    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:52:57.498377 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.498350    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"]
Nov 02 01:52:57.500641 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:52:57.500611    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f90563c_5030_495b_83b5_de17d38a075e.slice/crio-81d5a4733b146498ed6c9d6ba25ea53de6faa6619a552155ace2c010479b6884 WatchSource:0}: Error finding container 81d5a4733b146498ed6c9d6ba25ea53de6faa6619a552155ace2c010479b6884: Status 404 returned error can't find the container with id 81d5a4733b146498ed6c9d6ba25ea53de6faa6619a552155ace2c010479b6884
Nov 02 01:52:57.868714 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.868677    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z" event={"ID":"4f90563c-5030-495b-83b5-de17d38a075e","Type":"ContainerStarted","Data":"d4dc57955ca6f951f94f3bab55e6e9bc77e5298b430e1fe5dcf8836b266214a4"}
Nov 02 01:52:57.868911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.868731    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z" event={"ID":"4f90563c-5030-495b-83b5-de17d38a075e","Type":"ContainerStarted","Data":"81d5a4733b146498ed6c9d6ba25ea53de6faa6619a552155ace2c010479b6884"}
Nov 02 01:52:57.889426 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:57.889367    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z" podStartSLOduration=1.889346714 podStartE2EDuration="1.889346714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:57.887454894 +0000 UTC m=+2283.192387720" watchObservedRunningTime="2026-11-02 01:52:57.889346714 +0000 UTC m=+2283.194279540"
Nov 02 01:52:58.875784 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:58.875748    4851 generic.go:356] "Generic (PLEG): container finished" podID="4f90563c-5030-495b-83b5-de17d38a075e" containerID="d4dc57955ca6f951f94f3bab55e6e9bc77e5298b430e1fe5dcf8836b266214a4" exitCode=0
Nov 02 01:52:58.876244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:52:58.875833    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z" event={"ID":"4f90563c-5030-495b-83b5-de17d38a075e","Type":"ContainerDied","Data":"d4dc57955ca6f951f94f3bab55e6e9bc77e5298b430e1fe5dcf8836b266214a4"}
Nov 02 01:53:00.019217 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.019193    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:53:00.118438 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.118413    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-openshift-ca\") pod \"4f90563c-5030-495b-83b5-de17d38a075e\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") "
Nov 02 01:53:00.118650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.118499    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f90563c-5030-495b-83b5-de17d38a075e-kube-api-access-r4vw6\" (UniqueName: \"kubernetes.io/projected/4f90563c-5030-495b-83b5-de17d38a075e-kube-api-access-r4vw6\") pod \"4f90563c-5030-495b-83b5-de17d38a075e\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") "
Nov 02 01:53:00.118650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.118566    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-xks-ca\") pod \"4f90563c-5030-495b-83b5-de17d38a075e\" (UID: \"4f90563c-5030-495b-83b5-de17d38a075e\") "
Nov 02 01:53:00.118836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.118812    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-openshift-ca" pod "4f90563c-5030-495b-83b5-de17d38a075e" (UID: "4f90563c-5030-495b-83b5-de17d38a075e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:00.120625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.120605    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-xks-ca" pod "4f90563c-5030-495b-83b5-de17d38a075e" (UID: "4f90563c-5030-495b-83b5-de17d38a075e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:00.120742 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.120720    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f90563c-5030-495b-83b5-de17d38a075e-kube-api-access-r4vw6" pod "4f90563c-5030-495b-83b5-de17d38a075e" (UID: "4f90563c-5030-495b-83b5-de17d38a075e"). InnerVolumeSpecName "kube-api-access-r4vw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:00.220125 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.220097    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:00.220125 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.220122    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f90563c-5030-495b-83b5-de17d38a075e-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:00.220313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.220131    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4vw6\" (UniqueName: \"kubernetes.io/projected/4f90563c-5030-495b-83b5-de17d38a075e-kube-api-access-r4vw6\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:00.887100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.887066    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z" event={"ID":"4f90563c-5030-495b-83b5-de17d38a075e","Type":"ContainerDied","Data":"81d5a4733b146498ed6c9d6ba25ea53de6faa6619a552155ace2c010479b6884"}
Nov 02 01:53:00.887261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.887119    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d5a4733b146498ed6c9d6ba25ea53de6faa6619a552155ace2c010479b6884"
Nov 02 01:53:00.887261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:00.887078    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"
Nov 02 01:53:03.341441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:03.341404    4851 scope.go:122] "RemoveContainer" containerID="7590d0a4fbe1a098893d3be0b47ad22d658e456d30e6d4ad32bf47f53cfece61"
Nov 02 01:53:03.353184 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:03.353164    4851 scope.go:122] "RemoveContainer" containerID="bfb21a432935dcc62b0372aecf91ba162d79b1ce8c03de17eb3682f40f8b5359"
Nov 02 01:53:11.032873 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:11.032771    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"]
Nov 02 01:53:11.036463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:11.036432    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e430dd-w7nd6"]
Nov 02 01:53:12.870318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:12.866390    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de2b849-a2a8-49fb-9a72-691a857f676a" path="/var/lib/kubelet/pods/5de2b849-a2a8-49fb-9a72-691a857f676a/volumes"
Nov 02 01:53:15.682090 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.682049    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"]
Nov 02 01:53:15.682495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.682444    4851 cpu_manager.go:414] "containerMap: removing container" podUID="4f90563c-5030-495b-83b5-de17d38a075e" containerName="revoke-keys"
Nov 02 01:53:15.682495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.682454    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f90563c-5030-495b-83b5-de17d38a075e" containerName="revoke-keys"
Nov 02 01:53:15.682572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.682546    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f90563c-5030-495b-83b5-de17d38a075e" containerName="revoke-keys"
Nov 02 01:53:15.687246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.687231    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:15.692744 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.692715    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:53:15.692893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.692801    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:15.692893 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.692858    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:53:15.693040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.692924    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:53:15.695730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.695706    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"]
Nov 02 01:53:15.816954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.816921    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjc4\" (UniqueName: \"kubernetes.io/projected/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-kube-api-access-frjc4\") pod \"maas-api-revoke-sub-79805ca9-jwb5k\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:15.817167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.816967    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jwb5k\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:15.817167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.817128    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jwb5k\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:15.918362 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.918324    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjc4\" (UniqueName: \"kubernetes.io/projected/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-kube-api-access-frjc4\") pod \"maas-api-revoke-sub-79805ca9-jwb5k\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:15.918584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.918372    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jwb5k\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:15.918584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.918436    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jwb5k\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:15.919089 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.919063    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jwb5k\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:15.921062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.921031    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jwb5k\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:15.930511 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:15.930487    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjc4\" (UniqueName: \"kubernetes.io/projected/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-kube-api-access-frjc4\") pod \"maas-api-revoke-sub-79805ca9-jwb5k\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:16.002115 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:16.002040    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:16.277014 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:16.276858    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"]
Nov 02 01:53:16.279407 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:53:16.279373    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea89348_3d6a_49e1_9a31_cb0e6cb5839b.slice/crio-96c15df4cedf315647566b0eb16f6ea68d7520f33e7d48f7cc39b5f17d7a7e59 WatchSource:0}: Error finding container 96c15df4cedf315647566b0eb16f6ea68d7520f33e7d48f7cc39b5f17d7a7e59: Status 404 returned error can't find the container with id 96c15df4cedf315647566b0eb16f6ea68d7520f33e7d48f7cc39b5f17d7a7e59
Nov 02 01:53:16.999311 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:16.999261    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k" event={"ID":"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b","Type":"ContainerStarted","Data":"aca90d98107e56287f2b0ed03cd88f982f16c5f80724653cc40942a34d55d841"}
Nov 02 01:53:16.999711 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:16.999331    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k" event={"ID":"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b","Type":"ContainerStarted","Data":"96c15df4cedf315647566b0eb16f6ea68d7520f33e7d48f7cc39b5f17d7a7e59"}
Nov 02 01:53:17.021231 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:17.021179    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k" podStartSLOduration=2.021165873 podStartE2EDuration="2.021165873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:17.017469988 +0000 UTC m=+2302.322402813" watchObservedRunningTime="2026-11-02 01:53:17.021165873 +0000 UTC m=+2302.326098699"
Nov 02 01:53:17.654561 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:53:17.654519    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:53:18.005399 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:18.005370    4851 generic.go:356] "Generic (PLEG): container finished" podID="5ea89348-3d6a-49e1-9a31-cb0e6cb5839b" containerID="aca90d98107e56287f2b0ed03cd88f982f16c5f80724653cc40942a34d55d841" exitCode=0
Nov 02 01:53:18.005796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:18.005462    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k" event={"ID":"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b","Type":"ContainerDied","Data":"aca90d98107e56287f2b0ed03cd88f982f16c5f80724653cc40942a34d55d841"}
Nov 02 01:53:19.151273 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:19.151248    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:19.254313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:19.254252    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-xks-ca\") pod \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") "
Nov 02 01:53:19.254495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:19.254410    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-openshift-ca\") pod \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") "
Nov 02 01:53:19.254495 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:19.254440    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-kube-api-access-frjc4\" (UniqueName: \"kubernetes.io/projected/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-kube-api-access-frjc4\") pod \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\" (UID: \"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b\") "
Nov 02 01:53:19.254758 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:19.254727    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-openshift-ca" pod "5ea89348-3d6a-49e1-9a31-cb0e6cb5839b" (UID: "5ea89348-3d6a-49e1-9a31-cb0e6cb5839b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:19.256546 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:19.256523    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-xks-ca" pod "5ea89348-3d6a-49e1-9a31-cb0e6cb5839b" (UID: "5ea89348-3d6a-49e1-9a31-cb0e6cb5839b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:19.256624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:19.256601    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-kube-api-access-frjc4" pod "5ea89348-3d6a-49e1-9a31-cb0e6cb5839b" (UID: "5ea89348-3d6a-49e1-9a31-cb0e6cb5839b"). InnerVolumeSpecName "kube-api-access-frjc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:19.355668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:19.355633    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:19.355668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:19.355663    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frjc4\" (UniqueName: \"kubernetes.io/projected/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-kube-api-access-frjc4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:19.355668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:19.355673    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:20.017872 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:20.017843    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"
Nov 02 01:53:20.018058 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:20.017879    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k" event={"ID":"5ea89348-3d6a-49e1-9a31-cb0e6cb5839b","Type":"ContainerDied","Data":"96c15df4cedf315647566b0eb16f6ea68d7520f33e7d48f7cc39b5f17d7a7e59"}
Nov 02 01:53:20.018058 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:20.017925    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96c15df4cedf315647566b0eb16f6ea68d7520f33e7d48f7cc39b5f17d7a7e59"
Nov 02 01:53:22.021960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:22.021925    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"]
Nov 02 01:53:22.024433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:22.024405    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e181c48-wr8jt"]
Nov 02 01:53:22.844504 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:22.844472    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999c26e2-095c-4233-8206-147d729428cd" path="/var/lib/kubelet/pods/999c26e2-095c-4233-8206-147d729428cd/volumes"
Nov 02 01:53:33.456245 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.456205    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"]
Nov 02 01:53:33.456676 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.456608    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5ea89348-3d6a-49e1-9a31-cb0e6cb5839b" containerName="revoke-keys"
Nov 02 01:53:33.456676 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.456620    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea89348-3d6a-49e1-9a31-cb0e6cb5839b" containerName="revoke-keys"
Nov 02 01:53:33.456756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.456684    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea89348-3d6a-49e1-9a31-cb0e6cb5839b" containerName="revoke-keys"
Nov 02 01:53:33.460145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.460128    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:33.464828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.464784    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:33.465960 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.465941    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:53:33.466125 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.466002    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:53:33.466193 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.466149    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:53:33.469405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.469386    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"]
Nov 02 01:53:33.531613 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.531581    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pn4n\" (UniqueName: \"kubernetes.io/projected/f834334a-1553-4a81-8838-8c5d3452a7a8-kube-api-access-2pn4n\") pod \"maas-api-revoke-sub-0b7e8edb-7kpws\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:33.531780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.531620    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kpws\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:33.531780 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.531643    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kpws\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:33.632221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.632187    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kpws\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:33.632409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.632281    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pn4n\" (UniqueName: \"kubernetes.io/projected/f834334a-1553-4a81-8838-8c5d3452a7a8-kube-api-access-2pn4n\") pod \"maas-api-revoke-sub-0b7e8edb-7kpws\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:33.632409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.632332    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kpws\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:33.632929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.632905    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kpws\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:33.634851 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.634827    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kpws\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:33.644423 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.644402    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pn4n\" (UniqueName: \"kubernetes.io/projected/f834334a-1553-4a81-8838-8c5d3452a7a8-kube-api-access-2pn4n\") pod \"maas-api-revoke-sub-0b7e8edb-7kpws\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:33.774205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:33.774144    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:34.052859 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:34.052781    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"]
Nov 02 01:53:34.056221 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:53:34.056191    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf834334a_1553_4a81_8838_8c5d3452a7a8.slice/crio-aa2420aec0ae91957e0f1ce7805583752a0e39b7c2ca7dad3a5e00f312a4a508 WatchSource:0}: Error finding container aa2420aec0ae91957e0f1ce7805583752a0e39b7c2ca7dad3a5e00f312a4a508: Status 404 returned error can't find the container with id aa2420aec0ae91957e0f1ce7805583752a0e39b7c2ca7dad3a5e00f312a4a508
Nov 02 01:53:34.120703 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:34.120673    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws" event={"ID":"f834334a-1553-4a81-8838-8c5d3452a7a8","Type":"ContainerStarted","Data":"aa2420aec0ae91957e0f1ce7805583752a0e39b7c2ca7dad3a5e00f312a4a508"}
Nov 02 01:53:35.128651 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:35.128569    4851 generic.go:356] "Generic (PLEG): container finished" podID="f834334a-1553-4a81-8838-8c5d3452a7a8" containerID="7ed5ac6edb3ac1dee4b5778f8c8692c8b0465820f549682bbe91c72984706fd9" exitCode=0
Nov 02 01:53:35.129014 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:35.128653    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws" event={"ID":"f834334a-1553-4a81-8838-8c5d3452a7a8","Type":"ContainerDied","Data":"7ed5ac6edb3ac1dee4b5778f8c8692c8b0465820f549682bbe91c72984706fd9"}
Nov 02 01:53:36.271132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:36.271106    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:36.353655 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:36.353624    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-xks-ca\") pod \"f834334a-1553-4a81-8838-8c5d3452a7a8\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") "
Nov 02 01:53:36.353854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:36.353697    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f834334a-1553-4a81-8838-8c5d3452a7a8-kube-api-access-2pn4n\" (UniqueName: \"kubernetes.io/projected/f834334a-1553-4a81-8838-8c5d3452a7a8-kube-api-access-2pn4n\") pod \"f834334a-1553-4a81-8838-8c5d3452a7a8\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") "
Nov 02 01:53:36.353854 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:36.353746    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-openshift-ca\") pod \"f834334a-1553-4a81-8838-8c5d3452a7a8\" (UID: \"f834334a-1553-4a81-8838-8c5d3452a7a8\") "
Nov 02 01:53:36.354128 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:36.354105    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-openshift-ca" pod "f834334a-1553-4a81-8838-8c5d3452a7a8" (UID: "f834334a-1553-4a81-8838-8c5d3452a7a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:36.355796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:36.355773    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-xks-ca" pod "f834334a-1553-4a81-8838-8c5d3452a7a8" (UID: "f834334a-1553-4a81-8838-8c5d3452a7a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:36.355919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:36.355906    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f834334a-1553-4a81-8838-8c5d3452a7a8-kube-api-access-2pn4n" pod "f834334a-1553-4a81-8838-8c5d3452a7a8" (UID: "f834334a-1553-4a81-8838-8c5d3452a7a8"). InnerVolumeSpecName "kube-api-access-2pn4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:36.454593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:36.454564    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pn4n\" (UniqueName: \"kubernetes.io/projected/f834334a-1553-4a81-8838-8c5d3452a7a8-kube-api-access-2pn4n\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:36.454593 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:36.454592    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:36.454838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:36.454601    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f834334a-1553-4a81-8838-8c5d3452a7a8-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:37.140337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:37.140232    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"
Nov 02 01:53:37.140337 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:37.140234    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws" event={"ID":"f834334a-1553-4a81-8838-8c5d3452a7a8","Type":"ContainerDied","Data":"aa2420aec0ae91957e0f1ce7805583752a0e39b7c2ca7dad3a5e00f312a4a508"}
Nov 02 01:53:37.140538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:37.140379    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa2420aec0ae91957e0f1ce7805583752a0e39b7c2ca7dad3a5e00f312a4a508"
Nov 02 01:53:44.160398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.160359    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"]
Nov 02 01:53:44.160938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.160764    4851 cpu_manager.go:414] "containerMap: removing container" podUID="f834334a-1553-4a81-8838-8c5d3452a7a8" containerName="revoke-keys"
Nov 02 01:53:44.160938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.160776    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f834334a-1553-4a81-8838-8c5d3452a7a8" containerName="revoke-keys"
Nov 02 01:53:44.160938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.160833    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="f834334a-1553-4a81-8838-8c5d3452a7a8" containerName="revoke-keys"
Nov 02 01:53:44.164065 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.164044    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.171484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.170971    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:53:44.171484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.171040    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:44.171484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.171044    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:53:44.171484 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.171087    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:53:44.174727 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.174694    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"]
Nov 02 01:53:44.219573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.219540    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j6g9s\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.219762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.219579    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j6g9s\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.219762 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.219600    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkh2\" (UniqueName: \"kubernetes.io/projected/b9b79fda-1358-4b22-ab66-63f2d5473ca3-kube-api-access-hnkh2\") pod \"maas-api-revoke-sub-3d4156f1-j6g9s\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.320609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.320578    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j6g9s\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.320783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.320618    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j6g9s\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.320783 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.320640    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkh2\" (UniqueName: \"kubernetes.io/projected/b9b79fda-1358-4b22-ab66-63f2d5473ca3-kube-api-access-hnkh2\") pod \"maas-api-revoke-sub-3d4156f1-j6g9s\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.321356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.321332    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j6g9s\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.323086 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.323067    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j6g9s\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.332909 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.332875    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkh2\" (UniqueName: \"kubernetes.io/projected/b9b79fda-1358-4b22-ab66-63f2d5473ca3-kube-api-access-hnkh2\") pod \"maas-api-revoke-sub-3d4156f1-j6g9s\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.479092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.479062    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:44.765630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:44.765598    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"]
Nov 02 01:53:44.769110 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:53:44.768674    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b79fda_1358_4b22_ab66_63f2d5473ca3.slice/crio-1c236ad2fb450363a47ca9059fd0d64ec83d05aa0b55270f2a32c86849ce498a WatchSource:0}: Error finding container 1c236ad2fb450363a47ca9059fd0d64ec83d05aa0b55270f2a32c86849ce498a: Status 404 returned error can't find the container with id 1c236ad2fb450363a47ca9059fd0d64ec83d05aa0b55270f2a32c86849ce498a
Nov 02 01:53:45.187938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:45.187900    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s" event={"ID":"b9b79fda-1358-4b22-ab66-63f2d5473ca3","Type":"ContainerStarted","Data":"15e8ce02d30812f1a2c38da845728a54d6e441113e3bf76f69f9c5e561284433"}
Nov 02 01:53:45.188351 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:45.187957    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s" event={"ID":"b9b79fda-1358-4b22-ab66-63f2d5473ca3","Type":"ContainerStarted","Data":"1c236ad2fb450363a47ca9059fd0d64ec83d05aa0b55270f2a32c86849ce498a"}
Nov 02 01:53:45.206908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:45.206863    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s" podStartSLOduration=1.206849499 podStartE2EDuration="1.206849499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:45.205071812 +0000 UTC m=+2330.510004638" watchObservedRunningTime="2026-11-02 01:53:45.206849499 +0000 UTC m=+2330.511782324"
Nov 02 01:53:46.194811 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:46.194776    4851 generic.go:356] "Generic (PLEG): container finished" podID="b9b79fda-1358-4b22-ab66-63f2d5473ca3" containerID="15e8ce02d30812f1a2c38da845728a54d6e441113e3bf76f69f9c5e561284433" exitCode=0
Nov 02 01:53:46.195248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:46.194863    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s" event={"ID":"b9b79fda-1358-4b22-ab66-63f2d5473ca3","Type":"ContainerDied","Data":"15e8ce02d30812f1a2c38da845728a54d6e441113e3bf76f69f9c5e561284433"}
Nov 02 01:53:47.339106 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:47.339080    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:47.451993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:47.451904    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-xks-ca\") pod \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") "
Nov 02 01:53:47.451993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:47.451942    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-openshift-ca\") pod \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") "
Nov 02 01:53:47.452214 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:47.452026    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b79fda-1358-4b22-ab66-63f2d5473ca3-kube-api-access-hnkh2\" (UniqueName: \"kubernetes.io/projected/b9b79fda-1358-4b22-ab66-63f2d5473ca3-kube-api-access-hnkh2\") pod \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\" (UID: \"b9b79fda-1358-4b22-ab66-63f2d5473ca3\") "
Nov 02 01:53:47.452360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:47.452337    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-openshift-ca" pod "b9b79fda-1358-4b22-ab66-63f2d5473ca3" (UID: "b9b79fda-1358-4b22-ab66-63f2d5473ca3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:47.454325 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:47.454300    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-xks-ca" pod "b9b79fda-1358-4b22-ab66-63f2d5473ca3" (UID: "b9b79fda-1358-4b22-ab66-63f2d5473ca3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:47.454483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:47.454460    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b79fda-1358-4b22-ab66-63f2d5473ca3-kube-api-access-hnkh2" pod "b9b79fda-1358-4b22-ab66-63f2d5473ca3" (UID: "b9b79fda-1358-4b22-ab66-63f2d5473ca3"). InnerVolumeSpecName "kube-api-access-hnkh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:47.553557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:47.553522    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:47.553557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:47.553553    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b79fda-1358-4b22-ab66-63f2d5473ca3-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:47.553557 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:47.553564    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkh2\" (UniqueName: \"kubernetes.io/projected/b9b79fda-1358-4b22-ab66-63f2d5473ca3-kube-api-access-hnkh2\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:53:47.655459 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:53:47.655425    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:53:48.206360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:48.206320    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s" event={"ID":"b9b79fda-1358-4b22-ab66-63f2d5473ca3","Type":"ContainerDied","Data":"1c236ad2fb450363a47ca9059fd0d64ec83d05aa0b55270f2a32c86849ce498a"}
Nov 02 01:53:48.206360 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:48.206354    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"
Nov 02 01:53:48.206553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:53:48.206377    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c236ad2fb450363a47ca9059fd0d64ec83d05aa0b55270f2a32c86849ce498a"
Nov 02 01:54:02.192457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.192421    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"]
Nov 02 01:54:02.197199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.197172    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j6g9s"]
Nov 02 01:54:02.217483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.217451    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"]
Nov 02 01:54:02.217906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.217888    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b9b79fda-1358-4b22-ab66-63f2d5473ca3" containerName="revoke-keys"
Nov 02 01:54:02.217998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.217908    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b79fda-1358-4b22-ab66-63f2d5473ca3" containerName="revoke-keys"
Nov 02 01:54:02.218070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.218052    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b79fda-1358-4b22-ab66-63f2d5473ca3" containerName="revoke-keys"
Nov 02 01:54:02.221197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.221174    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.229126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.229103    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:02.229254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.229167    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:54:02.229366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.229269    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:54:02.229433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.229365    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:54:02.231477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.231452    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"]
Nov 02 01:54:02.299929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.299895    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8fn\" (UniqueName: \"kubernetes.io/projected/003a4022-6a45-4883-adaa-3deda53dc592-kube-api-access-lt8fn\") pod \"maas-api-revoke-sub-3d4156f1-cxrv4\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.300132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.299957    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cxrv4\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.300218 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.300125    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cxrv4\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.400757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.400724    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cxrv4\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.400929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.400769    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8fn\" (UniqueName: \"kubernetes.io/projected/003a4022-6a45-4883-adaa-3deda53dc592-kube-api-access-lt8fn\") pod \"maas-api-revoke-sub-3d4156f1-cxrv4\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.400929 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.400824    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cxrv4\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.401581 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.401562    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cxrv4\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.403264 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.403245    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cxrv4\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.413315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.413268    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8fn\" (UniqueName: \"kubernetes.io/projected/003a4022-6a45-4883-adaa-3deda53dc592-kube-api-access-lt8fn\") pod \"maas-api-revoke-sub-3d4156f1-cxrv4\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.536509 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.536425    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:02.812280 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.812256    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"]
Nov 02 01:54:02.814263 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:54:02.814236    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003a4022_6a45_4883_adaa_3deda53dc592.slice/crio-e422f985771f43e8e89531d1c9a4d9526b0ec72eb0608614021e3af3114bb79b WatchSource:0}: Error finding container e422f985771f43e8e89531d1c9a4d9526b0ec72eb0608614021e3af3114bb79b: Status 404 returned error can't find the container with id e422f985771f43e8e89531d1c9a4d9526b0ec72eb0608614021e3af3114bb79b
Nov 02 01:54:02.845943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:02.845918    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b79fda-1358-4b22-ab66-63f2d5473ca3" path="/var/lib/kubelet/pods/b9b79fda-1358-4b22-ab66-63f2d5473ca3/volumes"
Nov 02 01:54:03.328514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:03.328477    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4" event={"ID":"003a4022-6a45-4883-adaa-3deda53dc592","Type":"ContainerStarted","Data":"1e523df3a0b594e05a984d3c534847dec576be0f7150380671f581cca63a0609"}
Nov 02 01:54:03.328940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:03.328532    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4" event={"ID":"003a4022-6a45-4883-adaa-3deda53dc592","Type":"ContainerStarted","Data":"e422f985771f43e8e89531d1c9a4d9526b0ec72eb0608614021e3af3114bb79b"}
Nov 02 01:54:03.348178 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:03.348129    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4" podStartSLOduration=1.348117875 podStartE2EDuration="1.348117875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:03.34662687 +0000 UTC m=+2348.651559696" watchObservedRunningTime="2026-11-02 01:54:03.348117875 +0000 UTC m=+2348.653050739"
Nov 02 01:54:04.334953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:04.334913    4851 generic.go:356] "Generic (PLEG): container finished" podID="003a4022-6a45-4883-adaa-3deda53dc592" containerID="1e523df3a0b594e05a984d3c534847dec576be0f7150380671f581cca63a0609" exitCode=0
Nov 02 01:54:04.335387 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:04.334993    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4" event={"ID":"003a4022-6a45-4883-adaa-3deda53dc592","Type":"ContainerDied","Data":"1e523df3a0b594e05a984d3c534847dec576be0f7150380671f581cca63a0609"}
Nov 02 01:54:05.483848 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:05.483824    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:05.529144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:05.529117    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-xks-ca\") pod \"003a4022-6a45-4883-adaa-3deda53dc592\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") "
Nov 02 01:54:05.529318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:05.529188    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-openshift-ca\") pod \"003a4022-6a45-4883-adaa-3deda53dc592\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") "
Nov 02 01:54:05.529318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:05.529274    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003a4022-6a45-4883-adaa-3deda53dc592-kube-api-access-lt8fn\" (UniqueName: \"kubernetes.io/projected/003a4022-6a45-4883-adaa-3deda53dc592-kube-api-access-lt8fn\") pod \"003a4022-6a45-4883-adaa-3deda53dc592\" (UID: \"003a4022-6a45-4883-adaa-3deda53dc592\") "
Nov 02 01:54:05.529621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:05.529596    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-openshift-ca" pod "003a4022-6a45-4883-adaa-3deda53dc592" (UID: "003a4022-6a45-4883-adaa-3deda53dc592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:05.531379 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:05.531353    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-xks-ca" pod "003a4022-6a45-4883-adaa-3deda53dc592" (UID: "003a4022-6a45-4883-adaa-3deda53dc592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:05.531508 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:05.531457    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003a4022-6a45-4883-adaa-3deda53dc592-kube-api-access-lt8fn" pod "003a4022-6a45-4883-adaa-3deda53dc592" (UID: "003a4022-6a45-4883-adaa-3deda53dc592"). InnerVolumeSpecName "kube-api-access-lt8fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:05.630476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:05.630402    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8fn\" (UniqueName: \"kubernetes.io/projected/003a4022-6a45-4883-adaa-3deda53dc592-kube-api-access-lt8fn\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:05.630476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:05.630429    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:05.630476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:05.630440    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a4022-6a45-4883-adaa-3deda53dc592-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:06.347225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:06.347195    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"
Nov 02 01:54:06.347419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:06.347226    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4" event={"ID":"003a4022-6a45-4883-adaa-3deda53dc592","Type":"ContainerDied","Data":"e422f985771f43e8e89531d1c9a4d9526b0ec72eb0608614021e3af3114bb79b"}
Nov 02 01:54:06.347419 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:06.347270    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e422f985771f43e8e89531d1c9a4d9526b0ec72eb0608614021e3af3114bb79b"
Nov 02 01:54:13.032813 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:13.032775    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"]
Nov 02 01:54:13.035359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:13.035327    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"]
Nov 02 01:54:13.038006 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:13.037977    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4c104cf-7l4hk"]
Nov 02 01:54:13.040021 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:13.040000    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f59528-bln6z"]
Nov 02 01:54:14.845388 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:14.845355    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131cb34c-381c-4fdf-85fa-1ca7609a16dd" path="/var/lib/kubelet/pods/131cb34c-381c-4fdf-85fa-1ca7609a16dd/volumes"
Nov 02 01:54:14.845883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:14.845867    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd" path="/var/lib/kubelet/pods/2b0bf252-75d1-4bbf-a907-f4e09c2bd0bd/volumes"
Nov 02 01:54:17.654869 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:54:17.654833    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:54:27.461233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.461188    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"]
Nov 02 01:54:27.461716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.461609    4851 cpu_manager.go:414] "containerMap: removing container" podUID="003a4022-6a45-4883-adaa-3deda53dc592" containerName="revoke-keys"
Nov 02 01:54:27.461716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.461622    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003a4022-6a45-4883-adaa-3deda53dc592" containerName="revoke-keys"
Nov 02 01:54:27.461716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.461715    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="003a4022-6a45-4883-adaa-3deda53dc592" containerName="revoke-keys"
Nov 02 01:54:27.464846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.464825    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:27.473458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.473427    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:54:27.473614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.473427    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:27.473614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.473431    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:54:27.473614 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.473514    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:54:27.474906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.474882    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"]
Nov 02 01:54:27.597934 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.597891    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tzmgz\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:27.598163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.597975    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9cw\" (UniqueName: \"kubernetes.io/projected/e2510edc-a0d2-4922-a193-a4e3e34999e5-kube-api-access-4x9cw\") pod \"maas-api-revoke-sub-e2fc9c5b-tzmgz\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:27.598163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.598023    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tzmgz\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:27.699585 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.699538    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9cw\" (UniqueName: \"kubernetes.io/projected/e2510edc-a0d2-4922-a193-a4e3e34999e5-kube-api-access-4x9cw\") pod \"maas-api-revoke-sub-e2fc9c5b-tzmgz\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:27.699774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.699595    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tzmgz\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:27.699774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.699652    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tzmgz\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:27.700421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.700399    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tzmgz\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:27.702355 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.702334    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-tzmgz\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:27.712444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.712386    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9cw\" (UniqueName: \"kubernetes.io/projected/e2510edc-a0d2-4922-a193-a4e3e34999e5-kube-api-access-4x9cw\") pod \"maas-api-revoke-sub-e2fc9c5b-tzmgz\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:27.780249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:27.780205    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:28.022580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:28.022481    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"]
Nov 02 01:54:28.025363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:28.025338    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d4722a5-95dzf"]
Nov 02 01:54:28.057957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:28.057933    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"]
Nov 02 01:54:28.060034 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:54:28.060006    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2510edc_a0d2_4922_a193_a4e3e34999e5.slice/crio-a3ffe1a04425ed75f39ae945935d75d264c43a9452aa51c3fa754f380eb1cfef WatchSource:0}: Error finding container a3ffe1a04425ed75f39ae945935d75d264c43a9452aa51c3fa754f380eb1cfef: Status 404 returned error can't find the container with id a3ffe1a04425ed75f39ae945935d75d264c43a9452aa51c3fa754f380eb1cfef
Nov 02 01:54:28.500418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:28.500379    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz" event={"ID":"e2510edc-a0d2-4922-a193-a4e3e34999e5","Type":"ContainerStarted","Data":"75fb92a582c68d72f9caa37a7bd6eed9b46e580e636edeba4c6d4cbee4eda9ed"}
Nov 02 01:54:28.500921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:28.500437    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz" event={"ID":"e2510edc-a0d2-4922-a193-a4e3e34999e5","Type":"ContainerStarted","Data":"a3ffe1a04425ed75f39ae945935d75d264c43a9452aa51c3fa754f380eb1cfef"}
Nov 02 01:54:28.519013 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:28.518966    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz" podStartSLOduration=1.518954691 podStartE2EDuration="1.518954691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:28.517684788 +0000 UTC m=+2373.822617613" watchObservedRunningTime="2026-11-02 01:54:28.518954691 +0000 UTC m=+2373.823887518"
Nov 02 01:54:28.846068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:28.845991    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b403e6-e800-4423-b28b-4a4c2a7831ae" path="/var/lib/kubelet/pods/94b403e6-e800-4423-b28b-4a4c2a7831ae/volumes"
Nov 02 01:54:29.507062 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:29.507029    4851 generic.go:356] "Generic (PLEG): container finished" podID="e2510edc-a0d2-4922-a193-a4e3e34999e5" containerID="75fb92a582c68d72f9caa37a7bd6eed9b46e580e636edeba4c6d4cbee4eda9ed" exitCode=0
Nov 02 01:54:29.507491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:29.507109    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz" event={"ID":"e2510edc-a0d2-4922-a193-a4e3e34999e5","Type":"ContainerDied","Data":"75fb92a582c68d72f9caa37a7bd6eed9b46e580e636edeba4c6d4cbee4eda9ed"}
Nov 02 01:54:30.649529 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:30.649506    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:30.825609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:30.825527    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-openshift-ca\") pod \"e2510edc-a0d2-4922-a193-a4e3e34999e5\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") "
Nov 02 01:54:30.825609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:30.825572    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2510edc-a0d2-4922-a193-a4e3e34999e5-kube-api-access-4x9cw\" (UniqueName: \"kubernetes.io/projected/e2510edc-a0d2-4922-a193-a4e3e34999e5-kube-api-access-4x9cw\") pod \"e2510edc-a0d2-4922-a193-a4e3e34999e5\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") "
Nov 02 01:54:30.825841 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:30.825659    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-xks-ca\") pod \"e2510edc-a0d2-4922-a193-a4e3e34999e5\" (UID: \"e2510edc-a0d2-4922-a193-a4e3e34999e5\") "
Nov 02 01:54:30.825913 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:30.825886    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-openshift-ca" pod "e2510edc-a0d2-4922-a193-a4e3e34999e5" (UID: "e2510edc-a0d2-4922-a193-a4e3e34999e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:30.826024 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:30.826009    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:30.827768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:30.827745    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2510edc-a0d2-4922-a193-a4e3e34999e5-kube-api-access-4x9cw" pod "e2510edc-a0d2-4922-a193-a4e3e34999e5" (UID: "e2510edc-a0d2-4922-a193-a4e3e34999e5"). InnerVolumeSpecName "kube-api-access-4x9cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:30.827853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:30.827830    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-xks-ca" pod "e2510edc-a0d2-4922-a193-a4e3e34999e5" (UID: "e2510edc-a0d2-4922-a193-a4e3e34999e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:30.927168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:30.927137    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x9cw\" (UniqueName: \"kubernetes.io/projected/e2510edc-a0d2-4922-a193-a4e3e34999e5-kube-api-access-4x9cw\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:30.927168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:30.927166    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2510edc-a0d2-4922-a193-a4e3e34999e5-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:31.520249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:31.520222    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"
Nov 02 01:54:31.520502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:31.520255    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz" event={"ID":"e2510edc-a0d2-4922-a193-a4e3e34999e5","Type":"ContainerDied","Data":"a3ffe1a04425ed75f39ae945935d75d264c43a9452aa51c3fa754f380eb1cfef"}
Nov 02 01:54:31.520502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:31.520332    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ffe1a04425ed75f39ae945935d75d264c43a9452aa51c3fa754f380eb1cfef"
Nov 02 01:54:34.735963 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:34.735930    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"]
Nov 02 01:54:34.736391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:34.736122    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh" podUID="dad85307-4fc4-4708-8153-00e8a0279d76" containerName="discovery" containerID="cri-o://996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0" gracePeriod=30
Nov 02 01:54:35.001690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.001611    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:54:35.167465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.167432    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-ca-configmap\") pod \"dad85307-4fc4-4708-8153-00e8a0279d76\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") "
Nov 02 01:54:35.167650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.167472    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-dns-cert\") pod \"dad85307-4fc4-4708-8153-00e8a0279d76\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") "
Nov 02 01:54:35.167650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.167501    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-cacerts\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-cacerts\") pod \"dad85307-4fc4-4708-8153-00e8a0279d76\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") "
Nov 02 01:54:35.167650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.167547    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-kube-api-access-hrwn4\" (UniqueName: \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-kube-api-access-hrwn4\") pod \"dad85307-4fc4-4708-8153-00e8a0279d76\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") "
Nov 02 01:54:35.167650 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.167596    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad85307-4fc4-4708-8153-00e8a0279d76-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad85307-4fc4-4708-8153-00e8a0279d76-local-certs\") pod \"dad85307-4fc4-4708-8153-00e8a0279d76\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") "
Nov 02 01:54:35.167831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.167727    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-istio-token\" (UniqueName: \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-istio-token\") pod \"dad85307-4fc4-4708-8153-00e8a0279d76\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") "
Nov 02 01:54:35.167831 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.167766    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-kubeconfig\") pod \"dad85307-4fc4-4708-8153-00e8a0279d76\" (UID: \"dad85307-4fc4-4708-8153-00e8a0279d76\") "
Nov 02 01:54:35.167961 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.167940    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-ca-configmap" pod "dad85307-4fc4-4708-8153-00e8a0279d76" (UID: "dad85307-4fc4-4708-8153-00e8a0279d76"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:35.168118 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.168086    4851 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-ca-configmap\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:35.169968 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.169943    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-dns-cert" pod "dad85307-4fc4-4708-8153-00e8a0279d76" (UID: "dad85307-4fc4-4708-8153-00e8a0279d76"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:35.170232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.170203    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-kubeconfig" pod "dad85307-4fc4-4708-8153-00e8a0279d76" (UID: "dad85307-4fc4-4708-8153-00e8a0279d76"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:35.170446 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.170424    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad85307-4fc4-4708-8153-00e8a0279d76-local-certs" pod "dad85307-4fc4-4708-8153-00e8a0279d76" (UID: "dad85307-4fc4-4708-8153-00e8a0279d76"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:35.170538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.170462    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-kube-api-access-hrwn4" pod "dad85307-4fc4-4708-8153-00e8a0279d76" (UID: "dad85307-4fc4-4708-8153-00e8a0279d76"). InnerVolumeSpecName "kube-api-access-hrwn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:35.170606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.170549    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-cacerts" pod "dad85307-4fc4-4708-8153-00e8a0279d76" (UID: "dad85307-4fc4-4708-8153-00e8a0279d76"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:35.170606 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.170573    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-istio-token" pod "dad85307-4fc4-4708-8153-00e8a0279d76" (UID: "dad85307-4fc4-4708-8153-00e8a0279d76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:35.269221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.269133    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrwn4\" (UniqueName: \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-kube-api-access-hrwn4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:35.269221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.269165    4851 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad85307-4fc4-4708-8153-00e8a0279d76-local-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:35.269221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.269190    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad85307-4fc4-4708-8153-00e8a0279d76-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:35.269221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.269199    4851 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-kubeconfig\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:35.269221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.269209    4851 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-istio-csr-dns-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:35.269221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.269218    4851 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dad85307-4fc4-4708-8153-00e8a0279d76-cacerts\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:35.545053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.544957    4851 generic.go:356] "Generic (PLEG): container finished" podID="dad85307-4fc4-4708-8153-00e8a0279d76" containerID="996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0" exitCode=0
Nov 02 01:54:35.545053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.545014    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"
Nov 02 01:54:35.545053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.545020    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh" event={"ID":"dad85307-4fc4-4708-8153-00e8a0279d76","Type":"ContainerDied","Data":"996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0"}
Nov 02 01:54:35.545398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.545086    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh" event={"ID":"dad85307-4fc4-4708-8153-00e8a0279d76","Type":"ContainerDied","Data":"dd33a836d4db3f2e993bd53761f6b11df86e70609ad0943ddc1e8b40f4af1681"}
Nov 02 01:54:35.545398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.545112    4851 scope.go:122] "RemoveContainer" containerID="996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0"
Nov 02 01:54:35.556781 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.556758    4851 scope.go:122] "RemoveContainer" containerID="996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0"
Nov 02 01:54:35.557092 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:54:35.557063    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0\": container with ID starting with 996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0 not found: ID does not exist" containerID="996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0"
Nov 02 01:54:35.557181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.557092    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0"} err="failed to get container status \"996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0\": rpc error: code = NotFound desc = could not find container \"996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0\": container with ID starting with 996b7e7223baa355e1962f3bc4df1ba6fa5d18341cd4a28b117434f204d8e0b0 not found: ID does not exist"
Nov 02 01:54:35.568506 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.568478    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"]
Nov 02 01:54:35.572900 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:35.572877    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvzjh"]
Nov 02 01:54:36.021833 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:36.021785    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"]
Nov 02 01:54:36.024436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:36.024413    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046c5160-z82lf"]
Nov 02 01:54:36.844635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:36.844603    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf5f50e-a1f8-43ac-9e17-163776479428" path="/var/lib/kubelet/pods/4cf5f50e-a1f8-43ac-9e17-163776479428/volumes"
Nov 02 01:54:36.845026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:36.845013    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad85307-4fc4-4708-8153-00e8a0279d76" path="/var/lib/kubelet/pods/dad85307-4fc4-4708-8153-00e8a0279d76/volumes"
Nov 02 01:54:37.691385 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.691343    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"]
Nov 02 01:54:37.692123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.692102    4851 cpu_manager.go:414] "containerMap: removing container" podUID="e2510edc-a0d2-4922-a193-a4e3e34999e5" containerName="revoke-keys"
Nov 02 01:54:37.694363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.692126    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2510edc-a0d2-4922-a193-a4e3e34999e5" containerName="revoke-keys"
Nov 02 01:54:37.694363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.692163    4851 cpu_manager.go:414] "containerMap: removing container" podUID="dad85307-4fc4-4708-8153-00e8a0279d76" containerName="discovery"
Nov 02 01:54:37.694363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.692170    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad85307-4fc4-4708-8153-00e8a0279d76" containerName="discovery"
Nov 02 01:54:37.694363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.692260    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2510edc-a0d2-4922-a193-a4e3e34999e5" containerName="revoke-keys"
Nov 02 01:54:37.694363 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.692275    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad85307-4fc4-4708-8153-00e8a0279d76" containerName="discovery"
Nov 02 01:54:37.697248 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.697211    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:37.702652 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.702623    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:54:37.702828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.702733    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:54:37.702988 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.702948    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:37.703842 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.703820    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:54:37.703991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.703971    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"]
Nov 02 01:54:37.893052 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.893025    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwgj\" (UniqueName: \"kubernetes.io/projected/54f3139e-15f9-47d6-9842-3c673f9bd675-kube-api-access-6gwgj\") pod \"maas-api-revoke-sub-5a29efb3-bdkmd\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:37.893232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.893073    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-bdkmd\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:37.893232 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.893151    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-bdkmd\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:37.994008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.993927    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-bdkmd\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:37.994008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.993980    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-bdkmd\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:37.994266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.994133    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwgj\" (UniqueName: \"kubernetes.io/projected/54f3139e-15f9-47d6-9842-3c673f9bd675-kube-api-access-6gwgj\") pod \"maas-api-revoke-sub-5a29efb3-bdkmd\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:37.994782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.994757    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-bdkmd\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:37.996476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:37.996449    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-bdkmd\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:38.006307 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:38.006268    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwgj\" (UniqueName: \"kubernetes.io/projected/54f3139e-15f9-47d6-9842-3c673f9bd675-kube-api-access-6gwgj\") pod \"maas-api-revoke-sub-5a29efb3-bdkmd\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:38.015414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:38.015392    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:38.292756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:38.292726    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"]
Nov 02 01:54:38.294668 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:54:38.294638    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f3139e_15f9_47d6_9842_3c673f9bd675.slice/crio-91e14d1dd41dc9f3e16977fa6d6ae2bd698bcc79cb68711f19c858d1cb83d545 WatchSource:0}: Error finding container 91e14d1dd41dc9f3e16977fa6d6ae2bd698bcc79cb68711f19c858d1cb83d545: Status 404 returned error can't find the container with id 91e14d1dd41dc9f3e16977fa6d6ae2bd698bcc79cb68711f19c858d1cb83d545
Nov 02 01:54:38.565155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:38.565120    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd" event={"ID":"54f3139e-15f9-47d6-9842-3c673f9bd675","Type":"ContainerStarted","Data":"dda65216e5dfe175e421b17f3071d3423deac0e8a71d5cd49aa4bbb8823b7097"}
Nov 02 01:54:38.565333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:38.565175    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd" event={"ID":"54f3139e-15f9-47d6-9842-3c673f9bd675","Type":"ContainerStarted","Data":"91e14d1dd41dc9f3e16977fa6d6ae2bd698bcc79cb68711f19c858d1cb83d545"}
Nov 02 01:54:38.583639 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:38.583589    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd" podStartSLOduration=1.583576373 podStartE2EDuration="1.583576373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:38.582719312 +0000 UTC m=+2383.887652135" watchObservedRunningTime="2026-11-02 01:54:38.583576373 +0000 UTC m=+2383.888509198"
Nov 02 01:54:39.576261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:39.576209    4851 generic.go:356] "Generic (PLEG): container finished" podID="54f3139e-15f9-47d6-9842-3c673f9bd675" containerID="dda65216e5dfe175e421b17f3071d3423deac0e8a71d5cd49aa4bbb8823b7097" exitCode=0
Nov 02 01:54:39.577033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:39.576317    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd" event={"ID":"54f3139e-15f9-47d6-9842-3c673f9bd675","Type":"ContainerDied","Data":"dda65216e5dfe175e421b17f3071d3423deac0e8a71d5cd49aa4bbb8823b7097"}
Nov 02 01:54:41.457635 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.457600    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:41.499378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.492734    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f3139e-15f9-47d6-9842-3c673f9bd675-kube-api-access-6gwgj\" (UniqueName: \"kubernetes.io/projected/54f3139e-15f9-47d6-9842-3c673f9bd675-kube-api-access-6gwgj\") pod \"54f3139e-15f9-47d6-9842-3c673f9bd675\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") "
Nov 02 01:54:41.499378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.492813    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-xks-ca\") pod \"54f3139e-15f9-47d6-9842-3c673f9bd675\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") "
Nov 02 01:54:41.499378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.492880    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-openshift-ca\") pod \"54f3139e-15f9-47d6-9842-3c673f9bd675\" (UID: \"54f3139e-15f9-47d6-9842-3c673f9bd675\") "
Nov 02 01:54:41.513316 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.506967    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-openshift-ca" pod "54f3139e-15f9-47d6-9842-3c673f9bd675" (UID: "54f3139e-15f9-47d6-9842-3c673f9bd675"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:41.540317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.523827    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f3139e-15f9-47d6-9842-3c673f9bd675-kube-api-access-6gwgj" pod "54f3139e-15f9-47d6-9842-3c673f9bd675" (UID: "54f3139e-15f9-47d6-9842-3c673f9bd675"). InnerVolumeSpecName "kube-api-access-6gwgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:41.551317 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.540694    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-xks-ca" pod "54f3139e-15f9-47d6-9842-3c673f9bd675" (UID: "54f3139e-15f9-47d6-9842-3c673f9bd675"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:41.594201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.594113    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:41.594201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.594158    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f3139e-15f9-47d6-9842-3c673f9bd675-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:41.594201 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.594174    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gwgj\" (UniqueName: \"kubernetes.io/projected/54f3139e-15f9-47d6-9842-3c673f9bd675-kube-api-access-6gwgj\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:41.611786 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.611755    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"
Nov 02 01:54:41.611786 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.611769    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd" event={"ID":"54f3139e-15f9-47d6-9842-3c673f9bd675","Type":"ContainerDied","Data":"91e14d1dd41dc9f3e16977fa6d6ae2bd698bcc79cb68711f19c858d1cb83d545"}
Nov 02 01:54:41.612023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:41.611816    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91e14d1dd41dc9f3e16977fa6d6ae2bd698bcc79cb68711f19c858d1cb83d545"
Nov 02 01:54:47.655677 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:54:47.655642    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:54:53.615991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.615953    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"]
Nov 02 01:54:53.622466 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.622441    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cxrv4"]
Nov 02 01:54:53.641588 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.641552    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"]
Nov 02 01:54:53.641949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.641937    4851 cpu_manager.go:414] "containerMap: removing container" podUID="54f3139e-15f9-47d6-9842-3c673f9bd675" containerName="revoke-keys"
Nov 02 01:54:53.641994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.641950    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f3139e-15f9-47d6-9842-3c673f9bd675" containerName="revoke-keys"
Nov 02 01:54:53.642035 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.642025    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f3139e-15f9-47d6-9842-3c673f9bd675" containerName="revoke-keys"
Nov 02 01:54:53.645159 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.645143    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:53.651569 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.651539    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:54:53.652479 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.652378    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:54:53.652605 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.652588    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:54:53.653228 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.653204    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:53.657283 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.657259    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"]
Nov 02 01:54:53.712878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.712844    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7798\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:53.713095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.712975    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7798\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:53.713095 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.713020    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zk74\" (UniqueName: \"kubernetes.io/projected/d0afffe9-f037-4a7e-951e-5d99b7e0695e-kube-api-access-4zk74\") pod \"maas-api-revoke-sub-3d4156f1-s7798\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:53.814369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.814327    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7798\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:53.814369 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.814372    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zk74\" (UniqueName: \"kubernetes.io/projected/d0afffe9-f037-4a7e-951e-5d99b7e0695e-kube-api-access-4zk74\") pod \"maas-api-revoke-sub-3d4156f1-s7798\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:53.814663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.814445    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7798\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:53.815111 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.815089    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7798\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:53.816985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.816968    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7798\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:53.827257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.827235    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zk74\" (UniqueName: \"kubernetes.io/projected/d0afffe9-f037-4a7e-951e-5d99b7e0695e-kube-api-access-4zk74\") pod \"maas-api-revoke-sub-3d4156f1-s7798\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:53.960044 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:53.960011    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:54.234798 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:54.234768    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"]
Nov 02 01:54:54.236694 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:54:54.236668    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0afffe9_f037_4a7e_951e_5d99b7e0695e.slice/crio-e51a06ccab3e5ad2c98baca76b70a827eb8640d6864940463881dc7f7734cad4 WatchSource:0}: Error finding container e51a06ccab3e5ad2c98baca76b70a827eb8640d6864940463881dc7f7734cad4: Status 404 returned error can't find the container with id e51a06ccab3e5ad2c98baca76b70a827eb8640d6864940463881dc7f7734cad4
Nov 02 01:54:54.686212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:54.686177    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798" event={"ID":"d0afffe9-f037-4a7e-951e-5d99b7e0695e","Type":"ContainerStarted","Data":"339123ba796fcd548e584cc6e23d4db8b432f9d806ee8a3ad090f48b9eb9d4b0"}
Nov 02 01:54:54.686629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:54.686231    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798" event={"ID":"d0afffe9-f037-4a7e-951e-5d99b7e0695e","Type":"ContainerStarted","Data":"e51a06ccab3e5ad2c98baca76b70a827eb8640d6864940463881dc7f7734cad4"}
Nov 02 01:54:54.707528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:54.707475    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798" podStartSLOduration=1.707458849 podStartE2EDuration="1.707458849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:54.70488067 +0000 UTC m=+2400.009813497" watchObservedRunningTime="2026-11-02 01:54:54.707458849 +0000 UTC m=+2400.012391675"
Nov 02 01:54:54.846601 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:54.846524    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003a4022-6a45-4883-adaa-3deda53dc592" path="/var/lib/kubelet/pods/003a4022-6a45-4883-adaa-3deda53dc592/volumes"
Nov 02 01:54:55.693393 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:55.693356    4851 generic.go:356] "Generic (PLEG): container finished" podID="d0afffe9-f037-4a7e-951e-5d99b7e0695e" containerID="339123ba796fcd548e584cc6e23d4db8b432f9d806ee8a3ad090f48b9eb9d4b0" exitCode=0
Nov 02 01:54:55.693824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:55.693443    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798" event={"ID":"d0afffe9-f037-4a7e-951e-5d99b7e0695e","Type":"ContainerDied","Data":"339123ba796fcd548e584cc6e23d4db8b432f9d806ee8a3ad090f48b9eb9d4b0"}
Nov 02 01:54:56.423489 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:56.423459    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:54:56.425140 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:56.425113    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:54:56.842940 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:56.842911    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:56.940441 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:56.940408    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-openshift-ca\") pod \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") "
Nov 02 01:54:56.940649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:56.940542    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-xks-ca\") pod \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") "
Nov 02 01:54:56.940649 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:56.940563    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0afffe9-f037-4a7e-951e-5d99b7e0695e-kube-api-access-4zk74\" (UniqueName: \"kubernetes.io/projected/d0afffe9-f037-4a7e-951e-5d99b7e0695e-kube-api-access-4zk74\") pod \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\" (UID: \"d0afffe9-f037-4a7e-951e-5d99b7e0695e\") "
Nov 02 01:54:56.940794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:56.940770    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-openshift-ca" pod "d0afffe9-f037-4a7e-951e-5d99b7e0695e" (UID: "d0afffe9-f037-4a7e-951e-5d99b7e0695e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:56.942805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:56.942774    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-xks-ca" pod "d0afffe9-f037-4a7e-951e-5d99b7e0695e" (UID: "d0afffe9-f037-4a7e-951e-5d99b7e0695e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:56.942954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:56.942932    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0afffe9-f037-4a7e-951e-5d99b7e0695e-kube-api-access-4zk74" pod "d0afffe9-f037-4a7e-951e-5d99b7e0695e" (UID: "d0afffe9-f037-4a7e-951e-5d99b7e0695e"). InnerVolumeSpecName "kube-api-access-4zk74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:57.041261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:57.041175    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:57.041261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:57.041210    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0afffe9-f037-4a7e-951e-5d99b7e0695e-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:57.041261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:57.041220    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zk74\" (UniqueName: \"kubernetes.io/projected/d0afffe9-f037-4a7e-951e-5d99b7e0695e-kube-api-access-4zk74\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:54:57.706538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:57.706505    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"
Nov 02 01:54:57.706756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:57.706504    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798" event={"ID":"d0afffe9-f037-4a7e-951e-5d99b7e0695e","Type":"ContainerDied","Data":"e51a06ccab3e5ad2c98baca76b70a827eb8640d6864940463881dc7f7734cad4"}
Nov 02 01:54:57.706756 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:54:57.706626    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e51a06ccab3e5ad2c98baca76b70a827eb8640d6864940463881dc7f7734cad4"
Nov 02 01:55:03.387070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:03.387041    4851 scope.go:122] "RemoveContainer" containerID="1a0704d9a7ec48262526093972d4885c4cc440c688345711ff5c1f5e59a1a398"
Nov 02 01:55:03.396858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:03.396838    4851 scope.go:122] "RemoveContainer" containerID="68574e939f898f47340e1f0717d6fc4b92afd785d339e423368b5b7036b3858c"
Nov 02 01:55:03.405112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:03.405090    4851 scope.go:122] "RemoveContainer" containerID="1a691f5b8f3b8b1fe708bf3871177035dfae3d6ef28fcbac686d54eda66797b3"
Nov 02 01:55:03.413545 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:03.413526    4851 scope.go:122] "RemoveContainer" containerID="d65edc61c6146b2da8875fbd8609d99fd5e51e8b479ee936f37c1cba93071730"
Nov 02 01:55:03.421967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:03.421945    4851 scope.go:122] "RemoveContainer" containerID="c5ec9a80018c5beba6ff328fec864994986bbd8784fe2465fdd8817cb8621245"
Nov 02 01:55:03.430592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:03.430572    4851 scope.go:122] "RemoveContainer" containerID="e337b0e2a0ebc68456d47ee3830a15d4c551daf80590e5fd1d1e6a383ae46290"
Nov 02 01:55:12.444735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.444696    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"]
Nov 02 01:55:12.445416 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.445395    4851 cpu_manager.go:414] "containerMap: removing container" podUID="d0afffe9-f037-4a7e-951e-5d99b7e0695e" containerName="revoke-keys"
Nov 02 01:55:12.445512 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.445421    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0afffe9-f037-4a7e-951e-5d99b7e0695e" containerName="revoke-keys"
Nov 02 01:55:12.445572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.445512    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0afffe9-f037-4a7e-951e-5d99b7e0695e" containerName="revoke-keys"
Nov 02 01:55:12.450619 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.450590    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.454552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.454522    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-639dda-openshift-default-dockercfg-5h8tt\""
Nov 02 01:55:12.461503 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.461467    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"]
Nov 02 01:55:12.531717 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.531675    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-data\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.531717 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.531722    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-credential-socket\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.531964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.531820    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-envoy\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.531964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.531855    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-token\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.531964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.531874    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-podinfo\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.531964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.531955    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-certs\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.532105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.531980    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn5nw\" (UniqueName: \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-kube-api-access-vn5nw\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.532105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.532077    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-socket\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.532177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.532115    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istiod-ca-cert\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.634210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.634160    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-certs\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.634463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.634229    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn5nw\" (UniqueName: \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-kube-api-access-vn5nw\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.634463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.634327    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-socket\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.634463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.634358    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istiod-ca-cert\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.634463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.634395    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-data\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.634463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.634424    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-credential-socket\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.635319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.634480    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-envoy\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.635319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.634508    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-token\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.635319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.634535    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-podinfo\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.635819 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.635793    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istiod-ca-cert\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.635967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.635806    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-socket\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.636338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.636312    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-data\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.636883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.636839    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-certs\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.637061 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.636897    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-credential-socket\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.638392 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.638369    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-podinfo\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.639188 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.639164    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-envoy\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.655449 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.655420    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-token\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.655653 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.655603    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn5nw\" (UniqueName: \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-kube-api-access-vn5nw\") pod \"e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") " pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:12.767824 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:12.767728    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:13.059821 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:13.059788    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"]
Nov 02 01:55:13.061860 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:55:13.061835    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d7ca61_ecf7_45c9_9f73_5303d86cb811.slice/crio-d6e95797c72cdf8b92e7304ff671804346db96796d953ace565feb1c040c347f WatchSource:0}: Error finding container d6e95797c72cdf8b92e7304ff671804346db96796d953ace565feb1c040c347f: Status 404 returned error can't find the container with id d6e95797c72cdf8b92e7304ff671804346db96796d953ace565feb1c040c347f
Nov 02 01:55:13.064053 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:13.064025    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:55:13.064123 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:13.064091    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:55:13.064162 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:13.064125    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:55:13.834672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:13.834634    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" event={"ID":"c5d7ca61-ecf7-45c9-9f73-5303d86cb811","Type":"ContainerStarted","Data":"c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e"}
Nov 02 01:55:13.835102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:13.834721    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" event={"ID":"c5d7ca61-ecf7-45c9-9f73-5303d86cb811","Type":"ContainerStarted","Data":"d6e95797c72cdf8b92e7304ff671804346db96796d953ace565feb1c040c347f"}
Nov 02 01:55:13.860251 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:13.860198    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" podStartSLOduration=1.860184912 podStartE2EDuration="1.860184912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:13.857343924 +0000 UTC m=+2419.162276749" watchObservedRunningTime="2026-11-02 01:55:13.860184912 +0000 UTC m=+2419.165117962"
Nov 02 01:55:14.768198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:14.768162    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:14.773057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:14.773033    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:14.846804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:14.846773    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:14.847244 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:14.846851    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:17.233572 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.233535    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"]
Nov 02 01:55:17.237535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.237510    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.247033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.246997    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:55:17.247212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.247112    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:55:17.248906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.247282    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 01:55:17.248906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.248240    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-639dda-serving-cert\""
Nov 02 01:55:17.248906 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.248253    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8qwrd\""
Nov 02 01:55:17.251505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.251480    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"]
Nov 02 01:55:17.279144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.279094    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-metrics-certs\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.279366 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.279168    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6kr\" (UniqueName: \"kubernetes.io/projected/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-kube-api-access-lx6kr\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.279444 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.279380    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.279514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.279488    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.366953 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.366919    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"]
Nov 02 01:55:17.370576 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.370551    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:55:17.378721 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.378690    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-639dda\""
Nov 02 01:55:17.378980 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.378963    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-f2ca3bac-dockercfg-cswgh\""
Nov 02 01:55:17.380378 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.380352    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.380510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.380451    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-metrics-certs\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.380510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.380499    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6kr\" (UniqueName: \"kubernetes.io/projected/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-kube-api-access-lx6kr\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.380636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.380577    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.381386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.381361    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.382669 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.382644    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"]
Nov 02 01:55:17.383626 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.383600    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.383896 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.383874    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-metrics-certs\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.396170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.396143    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6kr\" (UniqueName: \"kubernetes.io/projected/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-kube-api-access-lx6kr\") pod \"maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.404418 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.404391    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"]
Nov 02 01:55:17.408657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.408637    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:55:17.422577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.422552    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"]
Nov 02 01:55:17.481800 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.481745    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6778204-394a-4fad-93dc-e2ef30b6bfc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv\" (UID: \"c6778204-394a-4fad-93dc-e2ef30b6bfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:55:17.482153 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.482126    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4jx\" (UniqueName: \"kubernetes.io/projected/c6778204-394a-4fad-93dc-e2ef30b6bfc3-kube-api-access-bl4jx\") pod \"payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv\" (UID: \"c6778204-394a-4fad-93dc-e2ef30b6bfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:55:17.482415 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.482381    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wndt9\" (UniqueName: \"kubernetes.io/projected/3837271c-0f47-48aa-8d53-ed05c8798d76-kube-api-access-wndt9\") pod \"payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth\" (UID: \"3837271c-0f47-48aa-8d53-ed05c8798d76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:55:17.482647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.482480    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3837271c-0f47-48aa-8d53-ed05c8798d76-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth\" (UID: \"3837271c-0f47-48aa-8d53-ed05c8798d76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:55:17.556971 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.556890    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:17.583779 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.583748    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndt9\" (UniqueName: \"kubernetes.io/projected/3837271c-0f47-48aa-8d53-ed05c8798d76-kube-api-access-wndt9\") pod \"payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth\" (UID: \"3837271c-0f47-48aa-8d53-ed05c8798d76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:55:17.583954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.583822    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3837271c-0f47-48aa-8d53-ed05c8798d76-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth\" (UID: \"3837271c-0f47-48aa-8d53-ed05c8798d76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:55:17.583954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.583866    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6778204-394a-4fad-93dc-e2ef30b6bfc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv\" (UID: \"c6778204-394a-4fad-93dc-e2ef30b6bfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:55:17.583954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.583933    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4jx\" (UniqueName: \"kubernetes.io/projected/c6778204-394a-4fad-93dc-e2ef30b6bfc3-kube-api-access-bl4jx\") pod \"payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv\" (UID: \"c6778204-394a-4fad-93dc-e2ef30b6bfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:55:17.584657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.584634    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6778204-394a-4fad-93dc-e2ef30b6bfc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv\" (UID: \"c6778204-394a-4fad-93dc-e2ef30b6bfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:55:17.585036 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.585011    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3837271c-0f47-48aa-8d53-ed05c8798d76-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth\" (UID: \"3837271c-0f47-48aa-8d53-ed05c8798d76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:55:17.595938 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.595912    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4jx\" (UniqueName: \"kubernetes.io/projected/c6778204-394a-4fad-93dc-e2ef30b6bfc3-kube-api-access-bl4jx\") pod \"payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv\" (UID: \"c6778204-394a-4fad-93dc-e2ef30b6bfc3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:55:17.596050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.595978    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wndt9\" (UniqueName: \"kubernetes.io/projected/3837271c-0f47-48aa-8d53-ed05c8798d76-kube-api-access-wndt9\") pod \"payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth\" (UID: \"3837271c-0f47-48aa-8d53-ed05c8798d76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:55:17.658578 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:55:17.658541    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:55:17.684205 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.684094    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:55:17.723846 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.723814    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:55:17.840943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.839562    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"]
Nov 02 01:55:17.850733 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.850696    4851 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:55:17.864243 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.864209    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg" event={"ID":"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438","Type":"ContainerStarted","Data":"bf1d203008a4510c5c2b2ebaf7b1eb2954f54c322d09e9fdf662dff52f2396b6"}
Nov 02 01:55:17.973318 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:17.973268    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"]
Nov 02 01:55:17.974703 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:55:17.974672    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6778204_394a_4fad_93dc_e2ef30b6bfc3.slice/crio-a9e627fe54e01380f8ab7ce367ed17cfc52cb35a50f518f22a08102775a24b98 WatchSource:0}: Error finding container a9e627fe54e01380f8ab7ce367ed17cfc52cb35a50f518f22a08102775a24b98: Status 404 returned error can't find the container with id a9e627fe54e01380f8ab7ce367ed17cfc52cb35a50f518f22a08102775a24b98
Nov 02 01:55:18.012754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.012716    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"]
Nov 02 01:55:18.025321 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:55:18.025272    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3837271c_0f47_48aa_8d53_ed05c8798d76.slice/crio-9bd3d0dc7dd3b29c308aad691bfb7b64c0c5058c221aa6328d9fed294ab14723 WatchSource:0}: Error finding container 9bd3d0dc7dd3b29c308aad691bfb7b64c0c5058c221aa6328d9fed294ab14723: Status 404 returned error can't find the container with id 9bd3d0dc7dd3b29c308aad691bfb7b64c0c5058c221aa6328d9fed294ab14723
Nov 02 01:55:18.870735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.870625    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv" event={"ID":"c6778204-394a-4fad-93dc-e2ef30b6bfc3","Type":"ContainerStarted","Data":"c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014"}
Nov 02 01:55:18.870735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.870697    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv" event={"ID":"c6778204-394a-4fad-93dc-e2ef30b6bfc3","Type":"ContainerStarted","Data":"a9e627fe54e01380f8ab7ce367ed17cfc52cb35a50f518f22a08102775a24b98"}
Nov 02 01:55:18.871221 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.870847    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:55:18.872326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.872274    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth" event={"ID":"3837271c-0f47-48aa-8d53-ed05c8798d76","Type":"ContainerStarted","Data":"534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b"}
Nov 02 01:55:18.872391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.872339    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth" event={"ID":"3837271c-0f47-48aa-8d53-ed05c8798d76","Type":"ContainerStarted","Data":"9bd3d0dc7dd3b29c308aad691bfb7b64c0c5058c221aa6328d9fed294ab14723"}
Nov 02 01:55:18.872391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.872362    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:55:18.873858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.873837    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg" event={"ID":"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438","Type":"ContainerStarted","Data":"8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f"}
Nov 02 01:55:18.873985 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.873971    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:18.889959 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.889907    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv" podStartSLOduration=1.889892475 podStartE2EDuration="1.889892475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:18.888180123 +0000 UTC m=+2424.193112949" watchObservedRunningTime="2026-11-02 01:55:18.889892475 +0000 UTC m=+2424.194825300"
Nov 02 01:55:18.908196 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.908128    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg" podStartSLOduration=1.310509373 podStartE2EDuration="1.908112942s" totalImagesPullingTime="597.603569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:18.907757446 +0000 UTC m=+2424.212690271" watchObservedRunningTime="2026-11-02 01:55:18.908112942 +0000 UTC m=+2424.213045769"
Nov 02 01:55:18.930389 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:18.930335    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth" podStartSLOduration=1.9303211249999999 podStartE2EDuration="1.930321125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:18.927666495 +0000 UTC m=+2424.232599322" watchObservedRunningTime="2026-11-02 01:55:18.930321125 +0000 UTC m=+2424.235253952"
Nov 02 01:55:21.022884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:21.022845    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"]
Nov 02 01:55:21.026590 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:21.026562    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kj96s"]
Nov 02 01:55:22.861774 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:22.861741    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769e37b8-624f-492c-9f56-dbd965280659" path="/var/lib/kubelet/pods/769e37b8-624f-492c-9f56-dbd965280659/volumes"
Nov 02 01:55:24.885040 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:24.885011    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:55:29.059612 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.059568    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"]
Nov 02 01:55:29.063120 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.063094    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.069825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.069801    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-639dda\"/\"kube-root-ca.crt\""
Nov 02 01:55:29.072424 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.072401    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-639dda\"/\"openshift-service-ca.crt\""
Nov 02 01:55:29.073836 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.073817    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-639dda\"/\"default-dockercfg-s67h2\""
Nov 02 01:55:29.074056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.074027    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-639dda\"/\"simulator-single-main-639dda-kserve-self-signed-certs\""
Nov 02 01:55:29.077579 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.077554    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"]
Nov 02 01:55:29.142068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.141971    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tmp-dir\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.142068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.142020    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-model-cache\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.142068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.142053    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wfb\" (UniqueName: \"kubernetes.io/projected/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-kube-api-access-68wfb\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.142343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.142176    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-home\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.142343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.142274    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tls-certs\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.142343 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.142325    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-dshm\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.250132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.243098    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tls-certs\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.250132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.243160    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-dshm\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.250132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.243228    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tmp-dir\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.250132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.243267    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-model-cache\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.250132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.243353    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wfb\" (UniqueName: \"kubernetes.io/projected/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-kube-api-access-68wfb\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.250132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.243392    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-home\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.250132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.243907    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-home\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.250132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.244607    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tmp-dir\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.250132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.244851    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-model-cache\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.252679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.252647    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tls-certs\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.252901 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.252850    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-dshm\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.259408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.259381    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wfb\" (UniqueName: \"kubernetes.io/projected/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-kube-api-access-68wfb\") pod \"simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") " pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.388349 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.388283    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:29.666860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.666829    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"]
Nov 02 01:55:29.668914 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:55:29.668890    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a4b8e33_98d6_4d5d_a907_6fe3ccb4cb08.slice/crio-f979b0a779fd5d9d98ed3f3a7828fd3f468a56a67435962ed5bf921165eaa6eb WatchSource:0}: Error finding container f979b0a779fd5d9d98ed3f3a7828fd3f468a56a67435962ed5bf921165eaa6eb: Status 404 returned error can't find the container with id f979b0a779fd5d9d98ed3f3a7828fd3f468a56a67435962ed5bf921165eaa6eb
Nov 02 01:55:29.844177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.844143    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.844373 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.844195    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 01:55:29.882600 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.882570    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:55:29.882853 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.882833    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:55:29.941015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.940927    4851 generic.go:356] "Generic (PLEG): container finished" podID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerID="c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e" exitCode=0
Nov 02 01:55:29.941199 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.941097    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" event={"ID":"c5d7ca61-ecf7-45c9-9f73-5303d86cb811","Type":"ContainerDied","Data":"c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e"}
Nov 02 01:55:29.941542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.941516    4851 scope.go:122] "RemoveContainer" containerID="c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e"
Nov 02 01:55:29.943458 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.943428    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:55:29.943580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.943470    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb" event={"ID":"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08","Type":"ContainerStarted","Data":"fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17"}
Nov 02 01:55:29.943580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.943503    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb" event={"ID":"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08","Type":"ContainerStarted","Data":"f979b0a779fd5d9d98ed3f3a7828fd3f468a56a67435962ed5bf921165eaa6eb"}
Nov 02 01:55:29.943580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.943510    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:55:29.943580 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.943549    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:55:29.986099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:29.986036    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb" podStartSLOduration=0.98601695 podStartE2EDuration="986.01695ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:29.984620727 +0000 UTC m=+2435.289553551" watchObservedRunningTime="2026-11-02 01:55:29.98601695 +0000 UTC m=+2435.290949778"
Nov 02 01:55:30.388643 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:30.388550    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:30.969033 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:30.968994    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" event={"ID":"c5d7ca61-ecf7-45c9-9f73-5303d86cb811","Type":"ContainerStarted","Data":"ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714"}
Nov 02 01:55:31.768223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:31.768186    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:31.768660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:31.768417    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.768660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:31.768468    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 01:55:32.768698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:32.768661    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.769124 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:32.768717    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 01:55:33.768398 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:33.768359    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 01:55:33.768599 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:33.768415    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 01:55:34.769017 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:34.768981    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.769478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:34.769037    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 01:55:35.768254 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:35.768219    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.768477 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:35.768281    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Nov 02 01:55:36.772676 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:36.772646    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:36.773132 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:36.773031    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:36.773859 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:36.773839    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:55:39.389323 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:39.389266    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:39.401771 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:39.401746    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:40.047198 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:55:40.047170    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:55:47.655362 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:55:47.655321    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:56:03.477764 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:03.477723    4851 scope.go:122] "RemoveContainer" containerID="2a1589b9b7be2f160465b7291a48c7481b107142189e62a0170d0b6fc15ee1a3"
Nov 02 01:56:03.486870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:03.486846    4851 scope.go:122] "RemoveContainer" containerID="d146bc6e9ddb9e9ef8a3faf62eb0068e35ee87fd9bccbb709112ca5b4ef89c03"
Nov 02 01:56:03.497273 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:03.497253    4851 scope.go:122] "RemoveContainer" containerID="6588dc74a0ec0099b05c47c522f20ab54eea07b3893c732a928b15ff38d05d68"
Nov 02 01:56:03.505533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:03.505514    4851 scope.go:122] "RemoveContainer" containerID="8014d174186ca9ac8e7399873f69922a4f19c0289f74ef6e1d80f097b6cb92ef"
Nov 02 01:56:03.522514 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:03.522460    4851 scope.go:122] "RemoveContainer" containerID="6b3268947707c3cdc83983e16419fa0de332807bbcb43510e26a3d2f8e4b688b"
Nov 02 01:56:03.533725 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:03.533699    4851 scope.go:122] "RemoveContainer" containerID="b0f95de92fafd49450a769e49c483225669c27acf26c815b89929e216e91854f"
Nov 02 01:56:13.704158 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:13.704073    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"]
Nov 02 01:56:15.034507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:15.034468    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"]
Nov 02 01:56:15.039143 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:15.039112    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pplt4"]
Nov 02 01:56:16.851679 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:16.851639    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3c6e6e-61a3-453d-aae4-72f9da7e2144" path="/var/lib/kubelet/pods/9e3c6e6e-61a3-453d-aae4-72f9da7e2144/volumes"
Nov 02 01:56:17.655770 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:56:17.655732    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:56:18.995757 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:18.995724    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"]
Nov 02 01:56:19.009103 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.009072    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"]
Nov 02 01:56:19.159060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.159022    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"]
Nov 02 01:56:19.159408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.159353    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg" podUID="7e5e4a05-5f56-4aa5-b1b4-f0eba799e438" containerName="maas-api" containerID="cri-o://8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f" gracePeriod=30
Nov 02 01:56:19.611990 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.611966    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:56:19.778798 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.778706    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-openshift-service-ca\") pod \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") "
Nov 02 01:56:19.778798 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.778776    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-metrics-certs\") pod \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") "
Nov 02 01:56:19.779057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.778846    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-kube-api-access-lx6kr\" (UniqueName: \"kubernetes.io/projected/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-kube-api-access-lx6kr\") pod \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") "
Nov 02 01:56:19.779057 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.778904    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-maas-api-tls\") pod \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\" (UID: \"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438\") "
Nov 02 01:56:19.779181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.779134    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-openshift-service-ca" pod "7e5e4a05-5f56-4aa5-b1b4-f0eba799e438" (UID: "7e5e4a05-5f56-4aa5-b1b4-f0eba799e438"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:19.781156 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.781124    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-metrics-certs" pod "7e5e4a05-5f56-4aa5-b1b4-f0eba799e438" (UID: "7e5e4a05-5f56-4aa5-b1b4-f0eba799e438"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:56:19.781284 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.781173    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-maas-api-tls" pod "7e5e4a05-5f56-4aa5-b1b4-f0eba799e438" (UID: "7e5e4a05-5f56-4aa5-b1b4-f0eba799e438"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:56:19.781284 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.781212    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-kube-api-access-lx6kr" pod "7e5e4a05-5f56-4aa5-b1b4-f0eba799e438" (UID: "7e5e4a05-5f56-4aa5-b1b4-f0eba799e438"). InnerVolumeSpecName "kube-api-access-lx6kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:19.880267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.880220    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx6kr\" (UniqueName: \"kubernetes.io/projected/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-kube-api-access-lx6kr\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.880267 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.880262    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-maas-api-tls\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.880533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.880277    4851 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-openshift-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:19.880533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:19.880320    4851 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438-metrics-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:20.267168 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:20.267130    4851 generic.go:356] "Generic (PLEG): container finished" podID="7e5e4a05-5f56-4aa5-b1b4-f0eba799e438" containerID="8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f" exitCode=0
Nov 02 01:56:20.267698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:20.267185    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"
Nov 02 01:56:20.267698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:20.267205    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg" event={"ID":"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438","Type":"ContainerDied","Data":"8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f"}
Nov 02 01:56:20.267698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:20.267259    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg" event={"ID":"7e5e4a05-5f56-4aa5-b1b4-f0eba799e438","Type":"ContainerDied","Data":"bf1d203008a4510c5c2b2ebaf7b1eb2954f54c322d09e9fdf662dff52f2396b6"}
Nov 02 01:56:20.267698 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:20.267280    4851 scope.go:122] "RemoveContainer" containerID="8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f"
Nov 02 01:56:20.277646 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:20.277627    4851 scope.go:122] "RemoveContainer" containerID="8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f"
Nov 02 01:56:20.277922 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:56:20.277902    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f\": container with ID starting with 8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f not found: ID does not exist" containerID="8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f"
Nov 02 01:56:20.277993 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:20.277935    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f"} err="failed to get container status \"8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f\": rpc error: code = NotFound desc = could not find container \"8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f\": container with ID starting with 8aa4ca7c515438a9f1823dbc0aeb82771f14fc7a6c55842f3ac63ba51965977f not found: ID does not exist"
Nov 02 01:56:20.290491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:20.290465    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"]
Nov 02 01:56:20.295002 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:20.294983    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-639dda-868765f5ff-25xdg"]
Nov 02 01:56:20.845230 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:20.845201    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5e4a05-5f56-4aa5-b1b4-f0eba799e438" path="/var/lib/kubelet/pods/7e5e4a05-5f56-4aa5-b1b4-f0eba799e438/volumes"
Nov 02 01:56:23.996223 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:23.996161    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth" podUID="3837271c-0f47-48aa-8d53-ed05c8798d76" containerName="payload-processing" containerID="cri-o://534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b" gracePeriod=25
Nov 02 01:56:24.009770 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:24.009724    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv" podUID="c6778204-394a-4fad-93dc-e2ef30b6bfc3" containerName="payload-pre-processing" containerID="cri-o://c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014" gracePeriod=25
Nov 02 01:56:26.222531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:26.222493    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"]
Nov 02 01:56:26.222998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:26.222690    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy" containerID="cri-o://ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714" gracePeriod=30
Nov 02 01:56:27.368747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.368707    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"]
Nov 02 01:56:27.374119 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.374090    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7798"]
Nov 02 01:56:27.394354 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.394310    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"]
Nov 02 01:56:27.394858 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.394838    4851 cpu_manager.go:414] "containerMap: removing container" podUID="7e5e4a05-5f56-4aa5-b1b4-f0eba799e438" containerName="maas-api"
Nov 02 01:56:27.394954 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.394859    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5e4a05-5f56-4aa5-b1b4-f0eba799e438" containerName="maas-api"
Nov 02 01:56:27.395025 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.394955    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5e4a05-5f56-4aa5-b1b4-f0eba799e438" containerName="maas-api"
Nov 02 01:56:27.399510 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.399486    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.403535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.403492    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:56:27.403719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.403579    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:56:27.404607 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.404579    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:56:27.404735 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.404670    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:56:27.408341 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.408313    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"]
Nov 02 01:56:27.449660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.449623    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b2dr\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.449827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.449690    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6p7\" (UniqueName: \"kubernetes.io/projected/2517908f-0c91-42cd-bcf4-f17afb7059e5-kube-api-access-5k6p7\") pod \"maas-api-revoke-sub-3d4156f1-4b2dr\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.449827 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.449754    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b2dr\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.551237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.551195    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6p7\" (UniqueName: \"kubernetes.io/projected/2517908f-0c91-42cd-bcf4-f17afb7059e5-kube-api-access-5k6p7\") pod \"maas-api-revoke-sub-3d4156f1-4b2dr\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.551237 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.551241    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b2dr\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.551550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.551351    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b2dr\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.552026 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.551998    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b2dr\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.553870 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.553840    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b2dr\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.563964 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.563938    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6p7\" (UniqueName: \"kubernetes.io/projected/2517908f-0c91-42cd-bcf4-f17afb7059e5-kube-api-access-5k6p7\") pod \"maas-api-revoke-sub-3d4156f1-4b2dr\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.715177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.715142    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:27.995236 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:27.995196    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"]
Nov 02 01:56:27.997342 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:56:27.997312    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2517908f_0c91_42cd_bcf4_f17afb7059e5.slice/crio-cb9d904f50936042957f7de12f6f026ca94833a413f6991761f41a5065d2b763 WatchSource:0}: Error finding container cb9d904f50936042957f7de12f6f026ca94833a413f6991761f41a5065d2b763: Status 404 returned error can't find the container with id cb9d904f50936042957f7de12f6f026ca94833a413f6991761f41a5065d2b763
Nov 02 01:56:28.311502 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:28.311465    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr" event={"ID":"2517908f-0c91-42cd-bcf4-f17afb7059e5","Type":"ContainerStarted","Data":"6378072c15e50698cb95012af133fd6fe3b7878f11c185a947ea2f8445d35674"}
Nov 02 01:56:28.311692 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:28.311518    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr" event={"ID":"2517908f-0c91-42cd-bcf4-f17afb7059e5","Type":"ContainerStarted","Data":"cb9d904f50936042957f7de12f6f026ca94833a413f6991761f41a5065d2b763"}
Nov 02 01:56:28.330340 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:28.330267    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr" podStartSLOduration=1.330247939 podStartE2EDuration="1.330247939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:28.32903146 +0000 UTC m=+2493.633964286" watchObservedRunningTime="2026-11-02 01:56:28.330247939 +0000 UTC m=+2493.635180764"
Nov 02 01:56:28.725163 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:28.725119    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb" podUID="0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" containerName="main" containerID="cri-o://fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17" gracePeriod=45
Nov 02 01:56:28.845166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:28.845137    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0afffe9-f037-4a7e-951e-5d99b7e0695e" path="/var/lib/kubelet/pods/d0afffe9-f037-4a7e-951e-5d99b7e0695e/volumes"
Nov 02 01:56:28.981185 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:28.981123    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:56:29.067146 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.067100    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-model-cache\") pod \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") "
Nov 02 01:56:29.067434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.067261    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-home\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-home\") pod \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") "
Nov 02 01:56:29.067434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.067366    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-dshm\") pod \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") "
Nov 02 01:56:29.067434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.067393    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-model-cache" pod "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" (UID: "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:29.067434 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.067424    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tls-certs\") pod \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") "
Nov 02 01:56:29.067675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.067456    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tmp-dir\") pod \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") "
Nov 02 01:56:29.067675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.067562    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-home" pod "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" (UID: "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:29.067675 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.067602    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-kube-api-access-68wfb\" (UniqueName: \"kubernetes.io/projected/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-kube-api-access-68wfb\") pod \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\" (UID: \"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08\") "
Nov 02 01:56:29.067845 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.067766    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tmp-dir" pod "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" (UID: "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:29.068043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.068019    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:29.068113 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.068053    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:29.068113 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.068070    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:29.069705 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.069682    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-dshm" pod "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" (UID: "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:29.070240 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.070219    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-kube-api-access-68wfb" pod "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" (UID: "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08"). InnerVolumeSpecName "kube-api-access-68wfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:29.070240 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.070218    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tls-certs" pod "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" (UID: "0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:56:29.169070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.169032    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:29.169070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.169063    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:29.169070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.169075    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68wfb\" (UniqueName: \"kubernetes.io/projected/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08-kube-api-access-68wfb\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:29.319137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.319046    4851 generic.go:356] "Generic (PLEG): container finished" podID="2517908f-0c91-42cd-bcf4-f17afb7059e5" containerID="6378072c15e50698cb95012af133fd6fe3b7878f11c185a947ea2f8445d35674" exitCode=0
Nov 02 01:56:29.319137 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.319092    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr" event={"ID":"2517908f-0c91-42cd-bcf4-f17afb7059e5","Type":"ContainerDied","Data":"6378072c15e50698cb95012af133fd6fe3b7878f11c185a947ea2f8445d35674"}
Nov 02 01:56:29.320465 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.320441    4851 generic.go:356] "Generic (PLEG): container finished" podID="0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" containerID="fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17" exitCode=0
Nov 02 01:56:29.320616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.320483    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"
Nov 02 01:56:29.320616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.320527    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb" event={"ID":"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08","Type":"ContainerDied","Data":"fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17"}
Nov 02 01:56:29.320616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.320572    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb" event={"ID":"0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08","Type":"ContainerDied","Data":"f979b0a779fd5d9d98ed3f3a7828fd3f468a56a67435962ed5bf921165eaa6eb"}
Nov 02 01:56:29.320616 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.320593    4851 scope.go:122] "RemoveContainer" containerID="fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17"
Nov 02 01:56:29.330370 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.330353    4851 scope.go:122] "RemoveContainer" containerID="fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17"
Nov 02 01:56:29.330617 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:56:29.330595    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17\": container with ID starting with fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17 not found: ID does not exist" containerID="fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17"
Nov 02 01:56:29.330683 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.330641    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17"} err="failed to get container status \"fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17\": rpc error: code = NotFound desc = could not find container \"fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17\": container with ID starting with fd9c0a0f4ceda424db46da2f5b787344b62bff33f72cac900e6f5bea59d23c17 not found: ID does not exist"
Nov 02 01:56:29.353048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.353023    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"]
Nov 02 01:56:29.356636 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.356614    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-639dda/simulator-single-main-639dda-kserve-79d4b9df7f-qjnbb"]
Nov 02 01:56:29.880794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.880760    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth" podUID="3837271c-0f47-48aa-8d53-ed05c8798d76" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:56:29.880794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:29.880771    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv" podUID="c6778204-394a-4fad-93dc-e2ef30b6bfc3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:56:30.462480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.462455    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:30.481382 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.481351    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2517908f-0c91-42cd-bcf4-f17afb7059e5-kube-api-access-5k6p7\" (UniqueName: \"kubernetes.io/projected/2517908f-0c91-42cd-bcf4-f17afb7059e5-kube-api-access-5k6p7\") pod \"2517908f-0c91-42cd-bcf4-f17afb7059e5\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") "
Nov 02 01:56:30.481553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.481447    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-xks-ca\") pod \"2517908f-0c91-42cd-bcf4-f17afb7059e5\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") "
Nov 02 01:56:30.481553 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.481535    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-openshift-ca\") pod \"2517908f-0c91-42cd-bcf4-f17afb7059e5\" (UID: \"2517908f-0c91-42cd-bcf4-f17afb7059e5\") "
Nov 02 01:56:30.481943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.481912    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-openshift-ca" pod "2517908f-0c91-42cd-bcf4-f17afb7059e5" (UID: "2517908f-0c91-42cd-bcf4-f17afb7059e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:30.484172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.484146    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-xks-ca" pod "2517908f-0c91-42cd-bcf4-f17afb7059e5" (UID: "2517908f-0c91-42cd-bcf4-f17afb7059e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:56:30.484326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.484195    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2517908f-0c91-42cd-bcf4-f17afb7059e5-kube-api-access-5k6p7" pod "2517908f-0c91-42cd-bcf4-f17afb7059e5" (UID: "2517908f-0c91-42cd-bcf4-f17afb7059e5"). InnerVolumeSpecName "kube-api-access-5k6p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:30.582481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.582394    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6p7\" (UniqueName: \"kubernetes.io/projected/2517908f-0c91-42cd-bcf4-f17afb7059e5-kube-api-access-5k6p7\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:30.582481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.582433    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:30.582481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.582444    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2517908f-0c91-42cd-bcf4-f17afb7059e5-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:30.844699 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:30.844620    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" path="/var/lib/kubelet/pods/0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08/volumes"
Nov 02 01:56:31.333257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.333227    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"
Nov 02 01:56:31.333257 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.333243    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr" event={"ID":"2517908f-0c91-42cd-bcf4-f17afb7059e5","Type":"ContainerDied","Data":"cb9d904f50936042957f7de12f6f026ca94833a413f6991761f41a5065d2b763"}
Nov 02 01:56:31.333706 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.333314    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb9d904f50936042957f7de12f6f026ca94833a413f6991761f41a5065d2b763"
Nov 02 01:56:31.469167 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.469146    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:56:31.490126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490060    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-kube-api-access-vn5nw\" (UniqueName: \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-kube-api-access-vn5nw\") pod \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") "
Nov 02 01:56:31.490126 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490094    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-socket\") pod \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") "
Nov 02 01:56:31.490353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490156    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-credential-socket\") pod \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") "
Nov 02 01:56:31.490353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490174    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-data\") pod \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") "
Nov 02 01:56:31.490353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490194    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istiod-ca-cert\") pod \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") "
Nov 02 01:56:31.490353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490224    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-token\" (UniqueName: \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-token\") pod \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") "
Nov 02 01:56:31.490353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490266    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-certs\") pod \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") "
Nov 02 01:56:31.490353 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490331    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-envoy\") pod \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") "
Nov 02 01:56:31.490687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490364    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-podinfo\") pod \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\" (UID: \"c5d7ca61-ecf7-45c9-9f73-5303d86cb811\") "
Nov 02 01:56:31.490687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490491    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-socket" pod "c5d7ca61-ecf7-45c9-9f73-5303d86cb811" (UID: "c5d7ca61-ecf7-45c9-9f73-5303d86cb811"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:31.490687 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490582    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-credential-socket" pod "c5d7ca61-ecf7-45c9-9f73-5303d86cb811" (UID: "c5d7ca61-ecf7-45c9-9f73-5303d86cb811"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:31.490840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490757    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-certs" pod "c5d7ca61-ecf7-45c9-9f73-5303d86cb811" (UID: "c5d7ca61-ecf7-45c9-9f73-5303d86cb811"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:31.490840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490757    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istiod-ca-cert" pod "c5d7ca61-ecf7-45c9-9f73-5303d86cb811" (UID: "c5d7ca61-ecf7-45c9-9f73-5303d86cb811"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:31.490840 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490784    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-data" pod "c5d7ca61-ecf7-45c9-9f73-5303d86cb811" (UID: "c5d7ca61-ecf7-45c9-9f73-5303d86cb811"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:31.490956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490839    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:31.490956 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.490858    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:31.492897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.492871    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-kube-api-access-vn5nw" pod "c5d7ca61-ecf7-45c9-9f73-5303d86cb811" (UID: "c5d7ca61-ecf7-45c9-9f73-5303d86cb811"). InnerVolumeSpecName "kube-api-access-vn5nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:31.493105 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.493084    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-token" pod "c5d7ca61-ecf7-45c9-9f73-5303d86cb811" (UID: "c5d7ca61-ecf7-45c9-9f73-5303d86cb811"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:31.493630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.493604    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-podinfo" pod "c5d7ca61-ecf7-45c9-9f73-5303d86cb811" (UID: "c5d7ca61-ecf7-45c9-9f73-5303d86cb811"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:56:31.494010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.493988    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-envoy" pod "c5d7ca61-ecf7-45c9-9f73-5303d86cb811" (UID: "c5d7ca61-ecf7-45c9-9f73-5303d86cb811"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:56:31.591530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.591501    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn5nw\" (UniqueName: \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-kube-api-access-vn5nw\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:31.591530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.591528    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:31.591754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.591540    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:31.591754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.591548    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:31.591754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.591557    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:31.591754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.591566    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:31.591754 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:31.591575    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d7ca61-ecf7-45c9-9f73-5303d86cb811-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:32.340253 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.340215    4851 generic.go:356] "Generic (PLEG): container finished" podID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerID="ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714" exitCode=0
Nov 02 01:56:32.340722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.340268    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"
Nov 02 01:56:32.340722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.340320    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" event={"ID":"c5d7ca61-ecf7-45c9-9f73-5303d86cb811","Type":"ContainerDied","Data":"ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714"}
Nov 02 01:56:32.340722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.340377    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4" event={"ID":"c5d7ca61-ecf7-45c9-9f73-5303d86cb811","Type":"ContainerDied","Data":"d6e95797c72cdf8b92e7304ff671804346db96796d953ace565feb1c040c347f"}
Nov 02 01:56:32.340722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.340400    4851 scope.go:122] "RemoveContainer" containerID="ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714"
Nov 02 01:56:32.350429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.350413    4851 scope.go:122] "RemoveContainer" containerID="c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e"
Nov 02 01:56:32.358657 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.358636    4851 scope.go:122] "RemoveContainer" containerID="ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714"
Nov 02 01:56:32.358920 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:56:32.358903    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714\": container with ID starting with ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714 not found: ID does not exist" containerID="ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714"
Nov 02 01:56:32.358970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.358928    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714"} err="failed to get container status \"ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714\": rpc error: code = NotFound desc = could not find container \"ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714\": container with ID starting with ad14158198da812683f1b4c8cea5a813efe9860a2ef9d3fb6708f53b818f0714 not found: ID does not exist"
Nov 02 01:56:32.358970 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.358943    4851 scope.go:122] "RemoveContainer" containerID="c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e"
Nov 02 01:56:32.359163 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:56:32.359148    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e\": container with ID starting with c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e not found: ID does not exist" containerID="c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e"
Nov 02 01:56:32.359206 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.359169    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e"} err="failed to get container status \"c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e\": rpc error: code = NotFound desc = could not find container \"c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e\": container with ID starting with c59a2fc05d514be2d5f7c9be06cb89c8b06bc3ea4ff020b0c5b4ec5c8ee6375e not found: ID does not exist"
Nov 02 01:56:32.362801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.362775    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"]
Nov 02 01:56:32.366778 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.366758    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-639dda-openshift-default-7d47fd67f7x28p4"]
Nov 02 01:56:32.845130 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:32.845099    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" path="/var/lib/kubelet/pods/c5d7ca61-ecf7-45c9-9f73-5303d86cb811/volumes"
Nov 02 01:56:34.023570 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:34.023540    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"]
Nov 02 01:56:34.029452 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:34.029420    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-ltzgv"]
Nov 02 01:56:34.845155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:34.845111    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f15e4a3-9a5b-40a6-b929-9b2e9ed00581" path="/var/lib/kubelet/pods/2f15e4a3-9a5b-40a6-b929-9b2e9ed00581/volumes"
Nov 02 01:56:39.911460 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:39.911414    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv" podUID="c6778204-394a-4fad-93dc-e2ef30b6bfc3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:56:39.915445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:39.915410    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth" podUID="3837271c-0f47-48aa-8d53-ed05c8798d76" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:56:44.268380 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.268357    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:56:44.271385 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.271365    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:56:44.334414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.334384    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3837271c-0f47-48aa-8d53-ed05c8798d76-kube-api-access-wndt9\" (UniqueName: \"kubernetes.io/projected/3837271c-0f47-48aa-8d53-ed05c8798d76-kube-api-access-wndt9\") pod \"3837271c-0f47-48aa-8d53-ed05c8798d76\" (UID: \"3837271c-0f47-48aa-8d53-ed05c8798d76\") "
Nov 02 01:56:44.334555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.334425    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6778204-394a-4fad-93dc-e2ef30b6bfc3-kube-api-access-bl4jx\" (UniqueName: \"kubernetes.io/projected/c6778204-394a-4fad-93dc-e2ef30b6bfc3-kube-api-access-bl4jx\") pod \"c6778204-394a-4fad-93dc-e2ef30b6bfc3\" (UID: \"c6778204-394a-4fad-93dc-e2ef30b6bfc3\") "
Nov 02 01:56:44.334555 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.334525    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6778204-394a-4fad-93dc-e2ef30b6bfc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6778204-394a-4fad-93dc-e2ef30b6bfc3-plugins-config-volume\") pod \"c6778204-394a-4fad-93dc-e2ef30b6bfc3\" (UID: \"c6778204-394a-4fad-93dc-e2ef30b6bfc3\") "
Nov 02 01:56:44.334719 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.334694    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3837271c-0f47-48aa-8d53-ed05c8798d76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3837271c-0f47-48aa-8d53-ed05c8798d76-plugins-config-volume\") pod \"3837271c-0f47-48aa-8d53-ed05c8798d76\" (UID: \"3837271c-0f47-48aa-8d53-ed05c8798d76\") "
Nov 02 01:56:44.335016 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.334988    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6778204-394a-4fad-93dc-e2ef30b6bfc3-plugins-config-volume" pod "c6778204-394a-4fad-93dc-e2ef30b6bfc3" (UID: "c6778204-394a-4fad-93dc-e2ef30b6bfc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:44.335125 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.334993    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3837271c-0f47-48aa-8d53-ed05c8798d76-plugins-config-volume" pod "3837271c-0f47-48aa-8d53-ed05c8798d76" (UID: "3837271c-0f47-48aa-8d53-ed05c8798d76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:44.336685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.336656    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3837271c-0f47-48aa-8d53-ed05c8798d76-kube-api-access-wndt9" pod "3837271c-0f47-48aa-8d53-ed05c8798d76" (UID: "3837271c-0f47-48aa-8d53-ed05c8798d76"). InnerVolumeSpecName "kube-api-access-wndt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:44.336685 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.336664    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6778204-394a-4fad-93dc-e2ef30b6bfc3-kube-api-access-bl4jx" pod "c6778204-394a-4fad-93dc-e2ef30b6bfc3" (UID: "c6778204-394a-4fad-93dc-e2ef30b6bfc3"). InnerVolumeSpecName "kube-api-access-bl4jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:44.432481 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.432446    4851 generic.go:356] "Generic (PLEG): container finished" podID="c6778204-394a-4fad-93dc-e2ef30b6bfc3" containerID="c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014" exitCode=0
Nov 02 01:56:44.432621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.432496    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"
Nov 02 01:56:44.432621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.432528    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv" event={"ID":"c6778204-394a-4fad-93dc-e2ef30b6bfc3","Type":"ContainerDied","Data":"c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014"}
Nov 02 01:56:44.432621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.432588    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv" event={"ID":"c6778204-394a-4fad-93dc-e2ef30b6bfc3","Type":"ContainerDied","Data":"a9e627fe54e01380f8ab7ce367ed17cfc52cb35a50f518f22a08102775a24b98"}
Nov 02 01:56:44.432621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.432618    4851 scope.go:122] "RemoveContainer" containerID="c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014"
Nov 02 01:56:44.434050 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.434022    4851 generic.go:356] "Generic (PLEG): container finished" podID="3837271c-0f47-48aa-8d53-ed05c8798d76" containerID="534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b" exitCode=0
Nov 02 01:56:44.434197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.434067    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"
Nov 02 01:56:44.434197 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.434068    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth" event={"ID":"3837271c-0f47-48aa-8d53-ed05c8798d76","Type":"ContainerDied","Data":"534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b"}
Nov 02 01:56:44.434420 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.434200    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth" event={"ID":"3837271c-0f47-48aa-8d53-ed05c8798d76","Type":"ContainerDied","Data":"9bd3d0dc7dd3b29c308aad691bfb7b64c0c5058c221aa6328d9fed294ab14723"}
Nov 02 01:56:44.435505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.435483    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6778204-394a-4fad-93dc-e2ef30b6bfc3-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:44.435505 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.435503    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3837271c-0f47-48aa-8d53-ed05c8798d76-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:44.435660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.435514    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wndt9\" (UniqueName: \"kubernetes.io/projected/3837271c-0f47-48aa-8d53-ed05c8798d76-kube-api-access-wndt9\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:44.435660 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.435523    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl4jx\" (UniqueName: \"kubernetes.io/projected/c6778204-394a-4fad-93dc-e2ef30b6bfc3-kube-api-access-bl4jx\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:56:44.441879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.441854    4851 scope.go:122] "RemoveContainer" containerID="c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014"
Nov 02 01:56:44.442154 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:56:44.442134    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014\": container with ID starting with c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014 not found: ID does not exist" containerID="c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014"
Nov 02 01:56:44.442233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.442166    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014"} err="failed to get container status \"c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014\": rpc error: code = NotFound desc = could not find container \"c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014\": container with ID starting with c6921c6f511cf2a29404fe12e009c4761c033b2b282d3effa3a274b9592e7014 not found: ID does not exist"
Nov 02 01:56:44.442233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.442201    4851 scope.go:122] "RemoveContainer" containerID="534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b"
Nov 02 01:56:44.453186 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.453168    4851 scope.go:122] "RemoveContainer" containerID="534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b"
Nov 02 01:56:44.453468 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:56:44.453449    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b\": container with ID starting with 534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b not found: ID does not exist" containerID="534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b"
Nov 02 01:56:44.453531 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.453474    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b"} err="failed to get container status \"534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b\": rpc error: code = NotFound desc = could not find container \"534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b\": container with ID starting with 534adf2d23d84cbfeaa727c0ce9bd9e90ca6acb21ff5cb5cd8fff20f48771d6b not found: ID does not exist"
Nov 02 01:56:44.458007 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.457986    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"]
Nov 02 01:56:44.462058 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.462038    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-639dda-5db85x2xvv"]
Nov 02 01:56:44.472621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.472602    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"]
Nov 02 01:56:44.478641 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.478622    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-639dda-5d6945cd5rsdth"]
Nov 02 01:56:44.845117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.845086    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3837271c-0f47-48aa-8d53-ed05c8798d76" path="/var/lib/kubelet/pods/3837271c-0f47-48aa-8d53-ed05c8798d76/volumes"
Nov 02 01:56:44.845492 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:44.845477    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6778204-394a-4fad-93dc-e2ef30b6bfc3" path="/var/lib/kubelet/pods/c6778204-394a-4fad-93dc-e2ef30b6bfc3/volumes"
Nov 02 01:56:45.033235 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:45.033204    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"]
Nov 02 01:56:45.035748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:45.035723    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-42pfv"]
Nov 02 01:56:46.845144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:56:46.845111    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b" path="/var/lib/kubelet/pods/b3f77fb0-bd5d-4e20-a0fc-bf5adb424c9b/volumes"
Nov 02 01:56:47.655404 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:56:47.655366    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:57:03.574995 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:57:03.574964    4851 scope.go:122] "RemoveContainer" containerID="6ee5e08b4f468dec0fd11fbab2864ccebfd7fe8dd963324c3afcccaae39e4b94"
Nov 02 01:57:11.022522 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:57:11.022481    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"]
Nov 02 01:57:11.024485 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:57:11.024463    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kv2vm"]
Nov 02 01:57:12.845542 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:57:12.845502    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88bc44b1-175a-4f62-8753-1b5ac9613bfc" path="/var/lib/kubelet/pods/88bc44b1-175a-4f62-8753-1b5ac9613bfc/volumes"
Nov 02 01:57:17.656011 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:57:17.655960    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:57:22.023010 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:57:22.022976    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"]
Nov 02 01:57:22.024794 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:57:22.024773    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-cs9hj"]
Nov 02 01:57:22.844809 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:57:22.844772    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf241d58-ffd7-44e1-af14-f30aed6c7c0c" path="/var/lib/kubelet/pods/bf241d58-ffd7-44e1-af14-f30aed6c7c0c/volumes"
Nov 02 01:57:42.031955 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:57:42.031872    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"]
Nov 02 01:57:42.034633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:57:42.034610    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-h96j9"]
Nov 02 01:57:42.845328 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:57:42.845271    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad9143a-341f-4bc6-aecb-356f948ed422" path="/var/lib/kubelet/pods/8ad9143a-341f-4bc6-aecb-356f948ed422/volumes"
Nov 02 01:57:47.655272 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:57:47.655236    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:58:01.021797 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:01.021762    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"]
Nov 02 01:58:01.027550 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:01.027525    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tq85z"]
Nov 02 01:58:02.846902 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:02.846863    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f90563c-5030-495b-83b5-de17d38a075e" path="/var/lib/kubelet/pods/4f90563c-5030-495b-83b5-de17d38a075e/volumes"
Nov 02 01:58:03.616022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:03.615990    4851 scope.go:122] "RemoveContainer" containerID="e3ab35d647cb081accc81e0dbd0d87715a002f01218f12aceb363e312d91214a"
Nov 02 01:58:03.626785 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:03.626749    4851 scope.go:122] "RemoveContainer" containerID="6739e8b10308f022ab189eb5c34fc285a5ab89971386e27a133c2022d86d0674"
Nov 02 01:58:03.634883 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:03.634860    4851 scope.go:122] "RemoveContainer" containerID="3608bc4f9fcfe78c44182551c8066ee58c94e5e7e25d0eb11b73879c478dac9b"
Nov 02 01:58:09.013903 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.013862    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"]
Nov 02 01:58:09.014338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014315    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c6778204-394a-4fad-93dc-e2ef30b6bfc3" containerName="payload-pre-processing"
Nov 02 01:58:09.014338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014329    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6778204-394a-4fad-93dc-e2ef30b6bfc3" containerName="payload-pre-processing"
Nov 02 01:58:09.014338 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014338    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy"
Nov 02 01:58:09.014457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014344    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy"
Nov 02 01:58:09.014457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014352    4851 cpu_manager.go:414] "containerMap: removing container" podUID="3837271c-0f47-48aa-8d53-ed05c8798d76" containerName="payload-processing"
Nov 02 01:58:09.014457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014356    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3837271c-0f47-48aa-8d53-ed05c8798d76" containerName="payload-processing"
Nov 02 01:58:09.014457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014373    4851 cpu_manager.go:414] "containerMap: removing container" podUID="2517908f-0c91-42cd-bcf4-f17afb7059e5" containerName="revoke-keys"
Nov 02 01:58:09.014457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014378    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2517908f-0c91-42cd-bcf4-f17afb7059e5" containerName="revoke-keys"
Nov 02 01:58:09.014457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014393    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" containerName="main"
Nov 02 01:58:09.014457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014399    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" containerName="main"
Nov 02 01:58:09.014684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014464    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6778204-394a-4fad-93dc-e2ef30b6bfc3" containerName="payload-pre-processing"
Nov 02 01:58:09.014684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014472    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy"
Nov 02 01:58:09.014684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014477    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="3837271c-0f47-48aa-8d53-ed05c8798d76" containerName="payload-processing"
Nov 02 01:58:09.014684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014484    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="2517908f-0c91-42cd-bcf4-f17afb7059e5" containerName="revoke-keys"
Nov 02 01:58:09.014684 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.014492    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4b8e33-98d6-4d5d-a907-6fe3ccb4cb08" containerName="main"
Nov 02 01:58:09.019190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.019172    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.023429 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.023396    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:58:09.024875 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.024850    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:58:09.025023 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.024937    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:58:09.026530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.026196    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"]
Nov 02 01:58:09.026530 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.025682    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:58:09.039457 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.039429    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbk7t\" (UniqueName: \"kubernetes.io/projected/8e23437f-204b-4a34-9615-83ac80c0bf7b-kube-api-access-lbk7t\") pod \"maas-api-revoke-sub-d576e68b-9pvjp\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.039625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.039481    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-9pvjp\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.039625 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.039500    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-9pvjp\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.140463 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.140423    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbk7t\" (UniqueName: \"kubernetes.io/projected/8e23437f-204b-4a34-9615-83ac80c0bf7b-kube-api-access-lbk7t\") pod \"maas-api-revoke-sub-d576e68b-9pvjp\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.140680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.140487    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-9pvjp\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.140680 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.140507    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-9pvjp\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.141109 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.141088    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-9pvjp\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.143147 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.143123    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-9pvjp\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.153148 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.153122    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbk7t\" (UniqueName: \"kubernetes.io/projected/8e23437f-204b-4a34-9615-83ac80c0bf7b-kube-api-access-lbk7t\") pod \"maas-api-revoke-sub-d576e68b-9pvjp\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.333166 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.333132    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:09.611910 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.611881    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"]
Nov 02 01:58:09.614505 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:58:09.614476    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e23437f_204b_4a34_9615_83ac80c0bf7b.slice/crio-de27924c4181942805666760b9b494484f5474814f9e3442fba421088d314675 WatchSource:0}: Error finding container de27924c4181942805666760b9b494484f5474814f9e3442fba421088d314675: Status 404 returned error can't find the container with id de27924c4181942805666760b9b494484f5474814f9e3442fba421088d314675
Nov 02 01:58:09.892169 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:09.892089    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp" event={"ID":"8e23437f-204b-4a34-9615-83ac80c0bf7b","Type":"ContainerStarted","Data":"de27924c4181942805666760b9b494484f5474814f9e3442fba421088d314675"}
Nov 02 01:58:10.898461 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:10.898374    4851 generic.go:356] "Generic (PLEG): container finished" podID="8e23437f-204b-4a34-9615-83ac80c0bf7b" containerID="1893af3e23e13676e5377ad486aee7cf7b517048e84a57e58c30a6c82e1ab650" exitCode=0
Nov 02 01:58:10.898848 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:10.898453    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp" event={"ID":"8e23437f-204b-4a34-9615-83ac80c0bf7b","Type":"ContainerDied","Data":"1893af3e23e13676e5377ad486aee7cf7b517048e84a57e58c30a6c82e1ab650"}
Nov 02 01:58:12.042440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.042412    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:12.067538 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.067497    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-xks-ca\") pod \"8e23437f-204b-4a34-9615-83ac80c0bf7b\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") "
Nov 02 01:58:12.067716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.067579    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-openshift-ca\") pod \"8e23437f-204b-4a34-9615-83ac80c0bf7b\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") "
Nov 02 01:58:12.067716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.067618    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e23437f-204b-4a34-9615-83ac80c0bf7b-kube-api-access-lbk7t\" (UniqueName: \"kubernetes.io/projected/8e23437f-204b-4a34-9615-83ac80c0bf7b-kube-api-access-lbk7t\") pod \"8e23437f-204b-4a34-9615-83ac80c0bf7b\" (UID: \"8e23437f-204b-4a34-9615-83ac80c0bf7b\") "
Nov 02 01:58:12.068008 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.067982    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-openshift-ca" pod "8e23437f-204b-4a34-9615-83ac80c0bf7b" (UID: "8e23437f-204b-4a34-9615-83ac80c0bf7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:58:12.070043 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.070021    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-xks-ca" pod "8e23437f-204b-4a34-9615-83ac80c0bf7b" (UID: "8e23437f-204b-4a34-9615-83ac80c0bf7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:58:12.070233 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.070211    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e23437f-204b-4a34-9615-83ac80c0bf7b-kube-api-access-lbk7t" pod "8e23437f-204b-4a34-9615-83ac80c0bf7b" (UID: "8e23437f-204b-4a34-9615-83ac80c0bf7b"). InnerVolumeSpecName "kube-api-access-lbk7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:58:12.168554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.168516    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbk7t\" (UniqueName: \"kubernetes.io/projected/8e23437f-204b-4a34-9615-83ac80c0bf7b-kube-api-access-lbk7t\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:58:12.168554 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.168556    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:58:12.168805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.168571    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e23437f-204b-4a34-9615-83ac80c0bf7b-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:58:12.910357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.910327    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"
Nov 02 01:58:12.910357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.910335    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp" event={"ID":"8e23437f-204b-4a34-9615-83ac80c0bf7b","Type":"ContainerDied","Data":"de27924c4181942805666760b9b494484f5474814f9e3442fba421088d314675"}
Nov 02 01:58:12.910568 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:12.910376    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de27924c4181942805666760b9b494484f5474814f9e3442fba421088d314675"
Nov 02 01:58:17.655676 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:58:17.655640    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:58:20.021173 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:20.021134    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"]
Nov 02 01:58:20.025523 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:20.025497    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jwb5k"]
Nov 02 01:58:20.845225 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:20.845189    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea89348-3d6a-49e1-9a31-cb0e6cb5839b" path="/var/lib/kubelet/pods/5ea89348-3d6a-49e1-9a31-cb0e6cb5839b/volumes"
Nov 02 01:58:27.871584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.871550    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"]
Nov 02 01:58:27.872022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.871955    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy"
Nov 02 01:58:27.872022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.871970    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy"
Nov 02 01:58:27.872022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.871999    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8e23437f-204b-4a34-9615-83ac80c0bf7b" containerName="revoke-keys"
Nov 02 01:58:27.872022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.872004    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e23437f-204b-4a34-9615-83ac80c0bf7b" containerName="revoke-keys"
Nov 02 01:58:27.872181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.872065    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d7ca61-ecf7-45c9-9f73-5303d86cb811" containerName="istio-proxy"
Nov 02 01:58:27.872181 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.872080    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e23437f-204b-4a34-9615-83ac80c0bf7b" containerName="revoke-keys"
Nov 02 01:58:27.875081 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.875062    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:27.883149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.883081    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:58:27.883149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.883139    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:58:27.883414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.883218    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:58:27.883414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.883335    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 01:58:27.885892 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.885865    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"]
Nov 02 01:58:27.955445 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.955410    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ks4s\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:27.955672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.955469    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ks4s\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:27.955672 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:27.955656    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdrkf\" (UniqueName: \"kubernetes.io/projected/228b1282-0bfe-4e7b-a865-60da3b5d48f7-kube-api-access-vdrkf\") pod \"maas-api-revoke-sub-d2bc93bc-6ks4s\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:28.056752 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:28.056718    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrkf\" (UniqueName: \"kubernetes.io/projected/228b1282-0bfe-4e7b-a865-60da3b5d48f7-kube-api-access-vdrkf\") pod \"maas-api-revoke-sub-d2bc93bc-6ks4s\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:28.056957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:28.056775    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ks4s\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:28.056957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:28.056804    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ks4s\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:28.057539 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:28.057519    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ks4s\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:28.059322 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:28.059305    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6ks4s\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:28.074099 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:28.074072    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrkf\" (UniqueName: \"kubernetes.io/projected/228b1282-0bfe-4e7b-a865-60da3b5d48f7-kube-api-access-vdrkf\") pod \"maas-api-revoke-sub-d2bc93bc-6ks4s\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:28.189914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:28.189880    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:28.469936 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:28.469912    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"]
Nov 02 01:58:28.472478 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:58:28.472452    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228b1282_0bfe_4e7b_a865_60da3b5d48f7.slice/crio-283de2a9e0f01cec1084d54a8f0f95b66dad50f16e20b8a72dde7134d30b2751 WatchSource:0}: Error finding container 283de2a9e0f01cec1084d54a8f0f95b66dad50f16e20b8a72dde7134d30b2751: Status 404 returned error can't find the container with id 283de2a9e0f01cec1084d54a8f0f95b66dad50f16e20b8a72dde7134d30b2751
Nov 02 01:58:29.045847 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:29.045746    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s" event={"ID":"228b1282-0bfe-4e7b-a865-60da3b5d48f7","Type":"ContainerStarted","Data":"97788da16bbeff60559db6f0b4e445d9e00130d6facdc18f2648dcae4c45d674"}
Nov 02 01:58:29.045847 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:29.045816    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s" event={"ID":"228b1282-0bfe-4e7b-a865-60da3b5d48f7","Type":"ContainerStarted","Data":"283de2a9e0f01cec1084d54a8f0f95b66dad50f16e20b8a72dde7134d30b2751"}
Nov 02 01:58:29.065715 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:29.065667    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s" podStartSLOduration=2.065652767 podStartE2EDuration="2.065652767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:29.064271322 +0000 UTC m=+2614.369204149" watchObservedRunningTime="2026-11-02 01:58:29.065652767 +0000 UTC m=+2614.370585591"
Nov 02 01:58:30.051969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:30.051932    4851 generic.go:356] "Generic (PLEG): container finished" podID="228b1282-0bfe-4e7b-a865-60da3b5d48f7" containerID="97788da16bbeff60559db6f0b4e445d9e00130d6facdc18f2648dcae4c45d674" exitCode=0
Nov 02 01:58:30.052425 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:30.052020    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s" event={"ID":"228b1282-0bfe-4e7b-a865-60da3b5d48f7","Type":"ContainerDied","Data":"97788da16bbeff60559db6f0b4e445d9e00130d6facdc18f2648dcae4c45d674"}
Nov 02 01:58:31.189928 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:31.189902    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:31.288602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:31.288557    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228b1282-0bfe-4e7b-a865-60da3b5d48f7-kube-api-access-vdrkf\" (UniqueName: \"kubernetes.io/projected/228b1282-0bfe-4e7b-a865-60da3b5d48f7-kube-api-access-vdrkf\") pod \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") "
Nov 02 01:58:31.288602 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:31.288605    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-xks-ca\") pod \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") "
Nov 02 01:58:31.288897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:31.288633    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-openshift-ca\") pod \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\" (UID: \"228b1282-0bfe-4e7b-a865-60da3b5d48f7\") "
Nov 02 01:58:31.289064 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:31.289038    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-openshift-ca" pod "228b1282-0bfe-4e7b-a865-60da3b5d48f7" (UID: "228b1282-0bfe-4e7b-a865-60da3b5d48f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:58:31.290998 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:31.290966    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-xks-ca" pod "228b1282-0bfe-4e7b-a865-60da3b5d48f7" (UID: "228b1282-0bfe-4e7b-a865-60da3b5d48f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:58:31.291112 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:31.291018    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228b1282-0bfe-4e7b-a865-60da3b5d48f7-kube-api-access-vdrkf" pod "228b1282-0bfe-4e7b-a865-60da3b5d48f7" (UID: "228b1282-0bfe-4e7b-a865-60da3b5d48f7"). InnerVolumeSpecName "kube-api-access-vdrkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:58:31.390075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:31.389983    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdrkf\" (UniqueName: \"kubernetes.io/projected/228b1282-0bfe-4e7b-a865-60da3b5d48f7-kube-api-access-vdrkf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:58:31.390075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:31.390015    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:58:31.390075 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:31.390028    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228b1282-0bfe-4e7b-a865-60da3b5d48f7-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:58:32.062867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:32.062834    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"
Nov 02 01:58:32.062867 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:32.062857    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s" event={"ID":"228b1282-0bfe-4e7b-a865-60da3b5d48f7","Type":"ContainerDied","Data":"283de2a9e0f01cec1084d54a8f0f95b66dad50f16e20b8a72dde7134d30b2751"}
Nov 02 01:58:32.063102 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:32.062910    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="283de2a9e0f01cec1084d54a8f0f95b66dad50f16e20b8a72dde7134d30b2751"
Nov 02 01:58:37.022911 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:37.022880    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"]
Nov 02 01:58:37.034310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:37.028359    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kpws"]
Nov 02 01:58:38.845792 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:38.845758    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f834334a-1553-4a81-8838-8c5d3452a7a8" path="/var/lib/kubelet/pods/f834334a-1553-4a81-8838-8c5d3452a7a8/volumes"
Nov 02 01:58:39.964048 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:39.964013    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"]
Nov 02 01:58:39.964662 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:39.964642    4851 cpu_manager.go:414] "containerMap: removing container" podUID="228b1282-0bfe-4e7b-a865-60da3b5d48f7" containerName="revoke-keys"
Nov 02 01:58:39.964709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:39.964667    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228b1282-0bfe-4e7b-a865-60da3b5d48f7" containerName="revoke-keys"
Nov 02 01:58:39.964822 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:39.964810    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="228b1282-0bfe-4e7b-a865-60da3b5d48f7" containerName="revoke-keys"
Nov 02 01:58:39.969850 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:39.969829    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:39.981517 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:39.981483    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"]
Nov 02 01:58:39.984574 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:39.984536    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-693506-openshift-default-dockercfg-htkrg\""
Nov 02 01:58:40.073182 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.073131    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661cbfc6-4689-4825-8aed-fc9a7ad59394-istiod-ca-cert\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.073430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.073202    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-data\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.073430 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.073321    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-certs\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.073549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.073434    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-envoy\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.073549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.073471    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-token\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.073549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.073500    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-credential-socket\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.073549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.073525    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-socket\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.073782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.073658    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftjs\" (UniqueName: \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-kube-api-access-tftjs\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.073782 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.073697    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-podinfo\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.174633 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.174588    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-envoy\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.174863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.174647    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-token\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.174863 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.174681    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-credential-socket\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.174848    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-socket\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.174910    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftjs\" (UniqueName: \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-kube-api-access-tftjs\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.174944    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-podinfo\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175015 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.174977    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661cbfc6-4689-4825-8aed-fc9a7ad59394-istiod-ca-cert\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.175024    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-data\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175270 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.175077    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-certs\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.175345    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-credential-socket\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175439 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.175424    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-socket\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.175480    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-data\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.175509    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-certs\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.175805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.175787    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661cbfc6-4689-4825-8aed-fc9a7ad59394-istiod-ca-cert\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.177306 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.177269    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-envoy\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.177647 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.177626    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-podinfo\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.187621 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.187595    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftjs\" (UniqueName: \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-kube-api-access-tftjs\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.187737 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.187688    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-token\") pod \"e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") " pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.284526 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.284433    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:40.564610 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.564584    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"]
Nov 02 01:58:40.567064 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:58:40.567039    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661cbfc6_4689_4825_8aed_fc9a7ad59394.slice/crio-8549c59ba8c6a6f69488f9bab1af1c4ca9eecf03f40b5dfe78ba4ae0e283c6e4 WatchSource:0}: Error finding container 8549c59ba8c6a6f69488f9bab1af1c4ca9eecf03f40b5dfe78ba4ae0e283c6e4: Status 404 returned error can't find the container with id 8549c59ba8c6a6f69488f9bab1af1c4ca9eecf03f40b5dfe78ba4ae0e283c6e4
Nov 02 01:58:40.569114 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.569084    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:58:40.569203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.569154    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:58:40.569203 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:40.569183    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:58:41.111952 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:41.111916    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" event={"ID":"661cbfc6-4689-4825-8aed-fc9a7ad59394","Type":"ContainerStarted","Data":"2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860"}
Nov 02 01:58:41.112433 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:41.111976    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" event={"ID":"661cbfc6-4689-4825-8aed-fc9a7ad59394","Type":"ContainerStarted","Data":"8549c59ba8c6a6f69488f9bab1af1c4ca9eecf03f40b5dfe78ba4ae0e283c6e4"}
Nov 02 01:58:41.137326 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:41.137258    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" podStartSLOduration=2.137244957 podStartE2EDuration="2.137244957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:41.135487802 +0000 UTC m=+2626.440420629" watchObservedRunningTime="2026-11-02 01:58:41.137244957 +0000 UTC m=+2626.442177782"
Nov 02 01:58:41.284734 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:41.284691    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:41.290022 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:41.289993    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:42.124949 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:42.124908    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:42.125969 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:42.125945    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:44.626642 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.626604    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"]
Nov 02 01:58:44.629609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.629586    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.646391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.646356    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:58:44.646551 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.646457    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:58:44.653386 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.653361    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-693506-serving-cert\""
Nov 02 01:58:44.653592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.653521    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8qwrd\""
Nov 02 01:58:44.653748 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.653689    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 01:58:44.657313 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.657256    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"]
Nov 02 01:58:44.718056 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.718024    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.718252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.718079    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-metrics-certs\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.718252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.718177    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.718252 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.718209    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfxvd\" (UniqueName: \"kubernetes.io/projected/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-kube-api-access-jfxvd\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.819709 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.819671    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-metrics-certs\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.819912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.819718    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.819912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.819755    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxvd\" (UniqueName: \"kubernetes.io/projected/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-kube-api-access-jfxvd\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.819912 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.819810    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.820521 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.820499    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.822376 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.822353    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-metrics-certs\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.822476 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.822392    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.831838 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.831813    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfxvd\" (UniqueName: \"kubernetes.io/projected/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-kube-api-access-jfxvd\") pod \"maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:44.856681 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.856649    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"]
Nov 02 01:58:44.859528 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.859510    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"]
Nov 02 01:58:44.859638 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.859621    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 01:58:44.862068 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.862049    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 01:58:44.869918 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.869891    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-693506\""
Nov 02 01:58:44.870117 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.870091    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c7228e9e-dockercfg-m4dzg\""
Nov 02 01:58:44.876842 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.876787    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"]
Nov 02 01:58:44.887826 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.887798    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"]
Nov 02 01:58:44.944668 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:44.944617    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:45.022897 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.022854    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a094bee7-3ba9-4b57-a857-995b218aad56-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb\" (UID: \"a094bee7-3ba9-4b57-a857-995b218aad56\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 01:58:45.023093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.023028    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mmgs\" (UniqueName: \"kubernetes.io/projected/a094bee7-3ba9-4b57-a857-995b218aad56-kube-api-access-9mmgs\") pod \"payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb\" (UID: \"a094bee7-3ba9-4b57-a857-995b218aad56\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 01:58:45.023189 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.023147    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9kng\" (UniqueName: \"kubernetes.io/projected/c9b114d5-8090-481a-b8a5-f41b05effa8b-kube-api-access-l9kng\") pod \"payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87\" (UID: \"c9b114d5-8090-481a-b8a5-f41b05effa8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 01:58:45.023271 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.023199    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b114d5-8090-481a-b8a5-f41b05effa8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87\" (UID: \"c9b114d5-8090-481a-b8a5-f41b05effa8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 01:58:45.124108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.124073    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a094bee7-3ba9-4b57-a857-995b218aad56-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb\" (UID: \"a094bee7-3ba9-4b57-a857-995b218aad56\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 01:58:45.124315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.124137    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mmgs\" (UniqueName: \"kubernetes.io/projected/a094bee7-3ba9-4b57-a857-995b218aad56-kube-api-access-9mmgs\") pod \"payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb\" (UID: \"a094bee7-3ba9-4b57-a857-995b218aad56\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 01:58:45.124315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.124182    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9kng\" (UniqueName: \"kubernetes.io/projected/c9b114d5-8090-481a-b8a5-f41b05effa8b-kube-api-access-l9kng\") pod \"payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87\" (UID: \"c9b114d5-8090-481a-b8a5-f41b05effa8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 01:58:45.124315 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.124205    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b114d5-8090-481a-b8a5-f41b05effa8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87\" (UID: \"c9b114d5-8090-481a-b8a5-f41b05effa8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 01:58:45.124917 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.124895    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b114d5-8090-481a-b8a5-f41b05effa8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87\" (UID: \"c9b114d5-8090-481a-b8a5-f41b05effa8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 01:58:45.124979 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.124906    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a094bee7-3ba9-4b57-a857-995b218aad56-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb\" (UID: \"a094bee7-3ba9-4b57-a857-995b218aad56\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 01:58:45.136747 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.136663    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9kng\" (UniqueName: \"kubernetes.io/projected/c9b114d5-8090-481a-b8a5-f41b05effa8b-kube-api-access-l9kng\") pod \"payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87\" (UID: \"c9b114d5-8090-481a-b8a5-f41b05effa8b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 01:58:45.136932 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.136863    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mmgs\" (UniqueName: \"kubernetes.io/projected/a094bee7-3ba9-4b57-a857-995b218aad56-kube-api-access-9mmgs\") pod \"payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb\" (UID: \"a094bee7-3ba9-4b57-a857-995b218aad56\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 01:58:45.174448 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.174416    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 01:58:45.185108 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.185056    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 01:58:45.253690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.253608    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"]
Nov 02 01:58:45.260274 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:58:45.259952    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dfc9a8_bc59_4b87_8ad1_8caaf7d31975.slice/crio-6a66bba76c5b7e7ab28d7c29e2edbc22f0e737cc315fbb4251ebd804d1993f70 WatchSource:0}: Error finding container 6a66bba76c5b7e7ab28d7c29e2edbc22f0e737cc315fbb4251ebd804d1993f70: Status 404 returned error can't find the container with id 6a66bba76c5b7e7ab28d7c29e2edbc22f0e737cc315fbb4251ebd804d1993f70
Nov 02 01:58:45.470150 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.469889    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"]
Nov 02 01:58:45.472721 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:58:45.472684    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b114d5_8090_481a_b8a5_f41b05effa8b.slice/crio-e9be3154a637ef1ba0b73df5385f4315a99dd774f18bfb70c092c5e13d879ec0 WatchSource:0}: Error finding container e9be3154a637ef1ba0b73df5385f4315a99dd774f18bfb70c092c5e13d879ec0: Status 404 returned error can't find the container with id e9be3154a637ef1ba0b73df5385f4315a99dd774f18bfb70c092c5e13d879ec0
Nov 02 01:58:45.481810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:45.481783    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"]
Nov 02 01:58:45.483827 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:58:45.483804    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda094bee7_3ba9_4b57_a857_995b218aad56.slice/crio-3b6bdc030dcc8ad88edb23c72d661754baf009cc3fdf477da405e689119e7ecd WatchSource:0}: Error finding container 3b6bdc030dcc8ad88edb23c72d661754baf009cc3fdf477da405e689119e7ecd: Status 404 returned error can't find the container with id 3b6bdc030dcc8ad88edb23c72d661754baf009cc3fdf477da405e689119e7ecd
Nov 02 01:58:46.147726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.147617    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx" event={"ID":"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975","Type":"ContainerStarted","Data":"c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa"}
Nov 02 01:58:46.147726 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.147721    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:46.148200 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.147745    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx" event={"ID":"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975","Type":"ContainerStarted","Data":"6a66bba76c5b7e7ab28d7c29e2edbc22f0e737cc315fbb4251ebd804d1993f70"}
Nov 02 01:58:46.149300 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.149255    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb" event={"ID":"a094bee7-3ba9-4b57-a857-995b218aad56","Type":"ContainerStarted","Data":"ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b"}
Nov 02 01:58:46.149421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.149325    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb" event={"ID":"a094bee7-3ba9-4b57-a857-995b218aad56","Type":"ContainerStarted","Data":"3b6bdc030dcc8ad88edb23c72d661754baf009cc3fdf477da405e689119e7ecd"}
Nov 02 01:58:46.149421 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.149350    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 01:58:46.150722 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.150703    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87" event={"ID":"c9b114d5-8090-481a-b8a5-f41b05effa8b","Type":"ContainerStarted","Data":"691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603"}
Nov 02 01:58:46.150820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.150743    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87" event={"ID":"c9b114d5-8090-481a-b8a5-f41b05effa8b","Type":"ContainerStarted","Data":"e9be3154a637ef1ba0b73df5385f4315a99dd774f18bfb70c092c5e13d879ec0"}
Nov 02 01:58:46.150820 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.150804    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 01:58:46.168732 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.168679    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx" podStartSLOduration=1.582596706 podStartE2EDuration="2.168665925s" totalImagesPullingTime="586.069219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:46.167552443 +0000 UTC m=+2631.472485271" watchObservedRunningTime="2026-11-02 01:58:46.168665925 +0000 UTC m=+2631.473598751"
Nov 02 01:58:46.185497 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.185439    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb" podStartSLOduration=2.185424772 podStartE2EDuration="2.185424772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:46.184641747 +0000 UTC m=+2631.489574587" watchObservedRunningTime="2026-11-02 01:58:46.185424772 +0000 UTC m=+2631.490357599"
Nov 02 01:58:46.204406 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:46.204354    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87" podStartSLOduration=2.204338505 podStartE2EDuration="2.204338505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:46.202475284 +0000 UTC m=+2631.507408111" watchObservedRunningTime="2026-11-02 01:58:46.204338505 +0000 UTC m=+2631.509271364"
Nov 02 01:58:47.661009 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:58:47.660975    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:58:52.167957 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:52.167928    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:58:54.934823 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:54.934778    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"]
Nov 02 01:58:54.942145 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:54.942117    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:54.952894 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:54.952863    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-693506\"/\"openshift-service-ca.crt\""
Nov 02 01:58:54.953077 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:54.952996    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-693506\"/\"default-dockercfg-zjvwz\""
Nov 02 01:58:54.954879 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:54.954851    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-693506\"/\"simulator-single-main-693506-kserve-self-signed-certs\""
Nov 02 01:58:54.955187 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:54.955166    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-693506\"/\"kube-root-ca.crt\""
Nov 02 01:58:54.959761 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:54.959348    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"]
Nov 02 01:58:55.029935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.029893    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-dshm\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.029935 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.029933    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-home\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.030175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.030049    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hxt\" (UniqueName: \"kubernetes.io/projected/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-kube-api-access-g8hxt\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.030175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.030086    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-model-cache\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.030175 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.030122    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tmp-dir\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.030365 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.030346    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tls-certs\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.131628 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.131581    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tls-certs\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.131849 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.131650    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-dshm\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.131849 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.131677    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-home\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.131849 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.131743    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hxt\" (UniqueName: \"kubernetes.io/projected/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-kube-api-access-g8hxt\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.132037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.131866    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-model-cache\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.132037 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.131937    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tmp-dir\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.132210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.132191    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-home\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.132319 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.132255    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-model-cache\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.132411 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.132388    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tmp-dir\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.134141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.134119    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-dshm\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.134847 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.134827    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tls-certs\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.145796 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.145761    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hxt\" (UniqueName: \"kubernetes.io/projected/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-kube-api-access-g8hxt\") pod \"simulator-single-main-693506-kserve-549f77b649-ktpsj\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") " pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.269155 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.269062    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:55.551951 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:55.551862    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"]
Nov 02 01:58:55.554995 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:58:55.554967    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod254f97e9_2ecf_4cd6_ba1f_44a5920b451a.slice/crio-d3873be7afb3bc5ef1c67e0c5440f77ea70efa054153ca0abf263e81175f36e3 WatchSource:0}: Error finding container d3873be7afb3bc5ef1c67e0c5440f77ea70efa054153ca0abf263e81175f36e3: Status 404 returned error can't find the container with id d3873be7afb3bc5ef1c67e0c5440f77ea70efa054153ca0abf263e81175f36e3
Nov 02 01:58:56.215694 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:56.215651    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj" event={"ID":"254f97e9-2ecf-4cd6-ba1f-44a5920b451a","Type":"ContainerStarted","Data":"27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe"}
Nov 02 01:58:56.216096 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:56.215702    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj" event={"ID":"254f97e9-2ecf-4cd6-ba1f-44a5920b451a","Type":"ContainerStarted","Data":"d3873be7afb3bc5ef1c67e0c5440f77ea70efa054153ca0abf263e81175f36e3"}
Nov 02 01:58:56.235828 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:56.235777    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj" podStartSLOduration=2.235762154 podStartE2EDuration="2.235762154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:56.23512735 +0000 UTC m=+2641.540060177" watchObservedRunningTime="2026-11-02 01:58:56.235762154 +0000 UTC m=+2641.540694979"
Nov 02 01:58:56.270190 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:56.270157    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:58:57.126595 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:57.126556    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Nov 02 01:58:57.126908 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:57.126627    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Nov 02 01:58:57.166361 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:57.166329    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 01:58:57.168210 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:57.168152    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 01:58:57.228592 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:57.228563    4851 generic.go:356] "Generic (PLEG): container finished" podID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerID="2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860" exitCode=0
Nov 02 01:58:57.229070 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:57.228636    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" event={"ID":"661cbfc6-4689-4825-8aed-fc9a7ad59394","Type":"ContainerDied","Data":"2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860"}
Nov 02 01:58:57.229150 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:57.229096    4851 scope.go:122] "RemoveContainer" containerID="2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860"
Nov 02 01:58:57.230437 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:57.230408    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:58:57.230533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:57.230484    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:58:57.230533 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:57.230518    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:58:58.261358 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:58.261316    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" event={"ID":"661cbfc6-4689-4825-8aed-fc9a7ad59394","Type":"ContainerStarted","Data":"c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe"}
Nov 02 01:58:58.285266 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:58.285229    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:58:58.285496 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:58.285473    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Nov 02 01:58:58.285558 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:58.285527    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Nov 02 01:58:59.284884 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:59.284849    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Nov 02 01:58:59.285312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:58:59.284899    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Nov 02 01:59:00.285459 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:00.285428    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Nov 02 01:59:00.285880 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:00.285484    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Nov 02 01:59:01.284852 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:01.284810    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Nov 02 01:59:01.285045 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:01.284874    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Nov 02 01:59:02.284947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:02.284916    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Nov 02 01:59:02.285409 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:02.284985    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Nov 02 01:59:03.284997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:03.284960    4851 patch_prober.go:29] interesting pod/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Nov 02 01:59:03.285436 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:03.285018    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Nov 02 01:59:03.662991 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:03.662960    4851 scope.go:122] "RemoveContainer" containerID="d4dc57955ca6f951f94f3bab55e6e9bc77e5298b430e1fe5dcf8836b266214a4"
Nov 02 01:59:03.671844 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:03.671820    4851 scope.go:122] "RemoveContainer" containerID="16787300ffb52e8a1742abe508c2ff0abc667ef459fbb666a53fb4dd789693d9"
Nov 02 01:59:03.680152 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:03.680132    4851 scope.go:122] "RemoveContainer" containerID="6360fba19ff58281191520bf39e571a8baa352c98e3774d4b5502f57ff02077c"
Nov 02 01:59:03.688491 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:03.688471    4851 scope.go:122] "RemoveContainer" containerID="52d0bf54ce3211ee224444220228f4676d4997638b695cb7d23bb87eb544b314"
Nov 02 01:59:04.289609 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:04.289583    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:59:04.290144 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:04.290119    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:59:04.291357 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:04.291338    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:59:05.269359 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:05.269244    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:59:05.282356 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:05.282330    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:59:05.310490 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:05.310451    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:59:17.655868 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:59:17.655829    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:59:31.023552 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:31.023516    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"]
Nov 02 01:59:31.025919 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:31.025899    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-tzmgz"]
Nov 02 01:59:32.850573 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:32.850539    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2510edc-a0d2-4922-a193-a4e3e34999e5" path="/var/lib/kubelet/pods/e2510edc-a0d2-4922-a193-a4e3e34999e5/volumes"
Nov 02 01:59:40.941730 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:40.941699    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"]
Nov 02 01:59:42.042715 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:42.042683    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"]
Nov 02 01:59:42.046804 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:42.046773    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-bdkmd"]
Nov 02 01:59:42.845584 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:42.845554    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f3139e-15f9-47d6-9842-3c673f9bd675" path="/var/lib/kubelet/pods/54f3139e-15f9-47d6-9842-3c673f9bd675/volumes"
Nov 02 01:59:46.243765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:46.243728    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"]
Nov 02 01:59:46.264540 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:46.264492    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"]
Nov 02 01:59:46.773451 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:46.773414    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"]
Nov 02 01:59:46.773947 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:46.773905    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx" podUID="b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975" containerName="maas-api" containerID="cri-o://c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa" gracePeriod=30
Nov 02 01:59:47.045127 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.045099    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:59:47.155624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.155594    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-openshift-service-ca\") pod \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") "
Nov 02 01:59:47.155817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.155676    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-maas-api-tls\") pod \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") "
Nov 02 01:59:47.155817 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.155773    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-kube-api-access-jfxvd\" (UniqueName: \"kubernetes.io/projected/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-kube-api-access-jfxvd\") pod \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") "
Nov 02 01:59:47.155943 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.155837    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-metrics-certs\") pod \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\" (UID: \"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975\") "
Nov 02 01:59:47.156039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.156016    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-openshift-service-ca" pod "b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975" (UID: "b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:47.156193 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.156177    4851 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-openshift-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:47.158031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.158008    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-maas-api-tls" pod "b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975" (UID: "b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:59:47.158031 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.158024    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-kube-api-access-jfxvd" pod "b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975" (UID: "b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975"). InnerVolumeSpecName "kube-api-access-jfxvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:47.158192 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.158103    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-metrics-certs" pod "b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975" (UID: "b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:59:47.256860 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.256825    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-maas-api-tls\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:47.257226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.256862    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfxvd\" (UniqueName: \"kubernetes.io/projected/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-kube-api-access-jfxvd\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:47.257226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.256880    4851 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975-metrics-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:47.531997 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.531962    4851 generic.go:356] "Generic (PLEG): container finished" podID="b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975" containerID="c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa" exitCode=0
Nov 02 01:59:47.532154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.532012    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"
Nov 02 01:59:47.532154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.532043    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx" event={"ID":"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975","Type":"ContainerDied","Data":"c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa"}
Nov 02 01:59:47.532154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.532103    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx" event={"ID":"b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975","Type":"ContainerDied","Data":"6a66bba76c5b7e7ab28d7c29e2edbc22f0e737cc315fbb4251ebd804d1993f70"}
Nov 02 01:59:47.532154 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.532131    4851 scope.go:122] "RemoveContainer" containerID="c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa"
Nov 02 01:59:47.541967 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.541951    4851 scope.go:122] "RemoveContainer" containerID="c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa"
Nov 02 01:59:47.542226 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:59:47.542206    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa\": container with ID starting with c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa not found: ID does not exist" containerID="c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa"
Nov 02 01:59:47.542310 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.542232    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa"} err="failed to get container status \"c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa\": rpc error: code = NotFound desc = could not find container \"c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa\": container with ID starting with c03ddadae2b1a671c0a1ca294e9210661e668e59bc2914473e6a5ccbefffd3fa not found: ID does not exist"
Nov 02 01:59:47.554663 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.554639    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"]
Nov 02 01:59:47.556414 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:47.556396    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-693506-7f8cd8ccf7-7dddx"]
Nov 02 01:59:47.655456 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:59:47.655415    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 01:59:48.844775 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:48.844745    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975" path="/var/lib/kubelet/pods/b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975/volumes"
Nov 02 01:59:51.244549 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:51.244511    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb" podUID="a094bee7-3ba9-4b57-a857-995b218aad56" containerName="payload-processing" containerID="cri-o://ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b" gracePeriod=25
Nov 02 01:59:51.264920 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:51.264887    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87" podUID="c9b114d5-8090-481a-b8a5-f41b05effa8b" containerName="payload-pre-processing" containerID="cri-o://691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603" gracePeriod=25
Nov 02 01:59:53.946921 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:53.946883    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"]
Nov 02 01:59:53.947507 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:53.947170    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy" containerID="cri-o://c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe" gracePeriod=30
Nov 02 01:59:55.963172 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:55.963136    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj" podUID="254f97e9-2ecf-4cd6-ba1f-44a5920b451a" containerName="main" containerID="cri-o://27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe" gracePeriod=45
Nov 02 01:59:56.302170 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.302143    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:59:56.455177 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455137    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tmp-dir\") pod \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") "
Nov 02 01:59:56.455412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455223    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-home\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-home\") pod \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") "
Nov 02 01:59:56.455412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455252    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tls-certs\" (UniqueName: \"kubernetes.io/secret/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tls-certs\") pod \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") "
Nov 02 01:59:56.455412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455321    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-dshm\") pod \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") "
Nov 02 01:59:56.455412 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455352    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-kube-api-access-g8hxt\" (UniqueName: \"kubernetes.io/projected/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-kube-api-access-g8hxt\") pod \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") "
Nov 02 01:59:56.455696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455414    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tmp-dir" pod "254f97e9-2ecf-4cd6-ba1f-44a5920b451a" (UID: "254f97e9-2ecf-4cd6-ba1f-44a5920b451a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:56.455696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455435    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-model-cache\") pod \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\" (UID: \"254f97e9-2ecf-4cd6-ba1f-44a5920b451a\") "
Nov 02 01:59:56.455696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455468    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-home" pod "254f97e9-2ecf-4cd6-ba1f-44a5920b451a" (UID: "254f97e9-2ecf-4cd6-ba1f-44a5920b451a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:56.455696 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455637    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-model-cache" pod "254f97e9-2ecf-4cd6-ba1f-44a5920b451a" (UID: "254f97e9-2ecf-4cd6-ba1f-44a5920b451a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:56.455937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455771    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:56.455937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455790    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:56.455937 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.455805    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:56.458081 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.458054    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-dshm" pod "254f97e9-2ecf-4cd6-ba1f-44a5920b451a" (UID: "254f97e9-2ecf-4cd6-ba1f-44a5920b451a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:56.458261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.458225    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-kube-api-access-g8hxt" pod "254f97e9-2ecf-4cd6-ba1f-44a5920b451a" (UID: "254f97e9-2ecf-4cd6-ba1f-44a5920b451a"). InnerVolumeSpecName "kube-api-access-g8hxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:56.458261 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.458234    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tls-certs" pod "254f97e9-2ecf-4cd6-ba1f-44a5920b451a" (UID: "254f97e9-2ecf-4cd6-ba1f-44a5920b451a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:59:56.458931 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.458910    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:59:56.461598 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.461575    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 01:59:56.556334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.556241    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:56.556334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.556271    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:56.556334 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.556283    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8hxt\" (UniqueName: \"kubernetes.io/projected/254f97e9-2ecf-4cd6-ba1f-44a5920b451a-kube-api-access-g8hxt\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:56.583342 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.583268    4851 generic.go:356] "Generic (PLEG): container finished" podID="254f97e9-2ecf-4cd6-ba1f-44a5920b451a" containerID="27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe" exitCode=0
Nov 02 01:59:56.583525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.583350    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj" event={"ID":"254f97e9-2ecf-4cd6-ba1f-44a5920b451a","Type":"ContainerDied","Data":"27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe"}
Nov 02 01:59:56.583525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.583375    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"
Nov 02 01:59:56.583525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.583411    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj" event={"ID":"254f97e9-2ecf-4cd6-ba1f-44a5920b451a","Type":"ContainerDied","Data":"d3873be7afb3bc5ef1c67e0c5440f77ea70efa054153ca0abf263e81175f36e3"}
Nov 02 01:59:56.583525 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.583434    4851 scope.go:122] "RemoveContainer" containerID="27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe"
Nov 02 01:59:56.594405 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.594278    4851 scope.go:122] "RemoveContainer" containerID="27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe"
Nov 02 01:59:56.594745 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:59:56.594722    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe\": container with ID starting with 27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe not found: ID does not exist" containerID="27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe"
Nov 02 01:59:56.594825 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.594755    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe"} err="failed to get container status \"27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe\": rpc error: code = NotFound desc = could not find container \"27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe\": container with ID starting with 27df8c10c10fd3818451c34e9340f177b8e099ea908e12ba114e7af6a999bcbe not found: ID does not exist"
Nov 02 01:59:56.608149 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.608122    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"]
Nov 02 01:59:56.612670 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.612647    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-693506/simulator-single-main-693506-kserve-549f77b649-ktpsj"]
Nov 02 01:59:56.845249 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:56.845152    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254f97e9-2ecf-4cd6-ba1f-44a5920b451a" path="/var/lib/kubelet/pods/254f97e9-2ecf-4cd6-ba1f-44a5920b451a/volumes"
Nov 02 01:59:57.163716 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:57.163674    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb" podUID="a094bee7-3ba9-4b57-a857-995b218aad56" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:59:57.164215 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:57.163712    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87" podUID="c9b114d5-8090-481a-b8a5-f41b05effa8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 01:59:58.846480 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.846445    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"]
Nov 02 01:59:58.846874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.846830    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975" containerName="maas-api"
Nov 02 01:59:58.846874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.846842    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975" containerName="maas-api"
Nov 02 01:59:58.846874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.846853    4851 cpu_manager.go:414] "containerMap: removing container" podUID="254f97e9-2ecf-4cd6-ba1f-44a5920b451a" containerName="main"
Nov 02 01:59:58.846874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.846858    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254f97e9-2ecf-4cd6-ba1f-44a5920b451a" containerName="main"
Nov 02 01:59:58.847039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.846946    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4dfc9a8-bc59-4b87-8ad1-8caaf7d31975" containerName="maas-api"
Nov 02 01:59:58.847039 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.846953    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="254f97e9-2ecf-4cd6-ba1f-44a5920b451a" containerName="main"
Nov 02 01:59:58.851535 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.851514    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:58.858207 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.857880    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-452a45db-kserve-self-signed-certs\""
Nov 02 01:59:58.860454 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.860432    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"]
Nov 02 01:59:58.980440 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.980373    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-home\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:58.980630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.980445    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tls-certs\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:58.980630 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.980588    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45n7p\" (UniqueName: \"kubernetes.io/projected/2c418ec3-c826-465a-bc5a-d73a9043ec8e-kube-api-access-45n7p\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:58.980777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.980645    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tmp-dir\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:58.980777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.980740    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-dshm\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:58.980777 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:58.980766    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-model-cache\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.081629 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.081596    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-dshm\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.081805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.081635    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-model-cache\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.081805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.081681    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-home\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.081805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.081709    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tls-certs\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.081805 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.081741    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45n7p\" (UniqueName: \"kubernetes.io/projected/2c418ec3-c826-465a-bc5a-d73a9043ec8e-kube-api-access-45n7p\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.082060 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.081866    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tmp-dir\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.082141 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.082118    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-model-cache\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.082333 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.082312    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-home\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.082408 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.082317    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tmp-dir\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.084624 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.084603    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-dshm\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.084790 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.084772    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tls-certs\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.095577 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.094916    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45n7p\" (UniqueName: \"kubernetes.io/projected/2c418ec3-c826-465a-bc5a-d73a9043ec8e-kube-api-access-45n7p\") pod \"e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") " pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.166499 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.166465    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 01:59:59.194874 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.194850    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:59:59.284478 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284447    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-credential-socket\") pod \"661cbfc6-4689-4825-8aed-fc9a7ad59394\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") "
Nov 02 01:59:59.284623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284537    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-token\" (UniqueName: \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-token\") pod \"661cbfc6-4689-4825-8aed-fc9a7ad59394\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") "
Nov 02 01:59:59.284623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284583    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-socket\") pod \"661cbfc6-4689-4825-8aed-fc9a7ad59394\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") "
Nov 02 01:59:59.284623 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284616    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661cbfc6-4689-4825-8aed-fc9a7ad59394-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661cbfc6-4689-4825-8aed-fc9a7ad59394-istiod-ca-cert\") pod \"661cbfc6-4689-4825-8aed-fc9a7ad59394\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") "
Nov 02 01:59:59.284768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284676    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-kube-api-access-tftjs\" (UniqueName: \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-kube-api-access-tftjs\") pod \"661cbfc6-4689-4825-8aed-fc9a7ad59394\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") "
Nov 02 01:59:59.284768 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284739    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-envoy\") pod \"661cbfc6-4689-4825-8aed-fc9a7ad59394\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") "
Nov 02 01:59:59.284878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284772    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-data\") pod \"661cbfc6-4689-4825-8aed-fc9a7ad59394\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") "
Nov 02 01:59:59.284878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284780    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-credential-socket" pod "661cbfc6-4689-4825-8aed-fc9a7ad59394" (UID: "661cbfc6-4689-4825-8aed-fc9a7ad59394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:59.284878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284798    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-certs\") pod \"661cbfc6-4689-4825-8aed-fc9a7ad59394\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") "
Nov 02 01:59:59.284878 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284849    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-podinfo\") pod \"661cbfc6-4689-4825-8aed-fc9a7ad59394\" (UID: \"661cbfc6-4689-4825-8aed-fc9a7ad59394\") "
Nov 02 01:59:59.285100 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.284900    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-socket" pod "661cbfc6-4689-4825-8aed-fc9a7ad59394" (UID: "661cbfc6-4689-4825-8aed-fc9a7ad59394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:59.285160 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.285037    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661cbfc6-4689-4825-8aed-fc9a7ad59394-istiod-ca-cert" pod "661cbfc6-4689-4825-8aed-fc9a7ad59394" (UID: "661cbfc6-4689-4825-8aed-fc9a7ad59394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:59.285246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.285124    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-data" pod "661cbfc6-4689-4825-8aed-fc9a7ad59394" (UID: "661cbfc6-4689-4825-8aed-fc9a7ad59394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:59.285246 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.285213    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-certs" pod "661cbfc6-4689-4825-8aed-fc9a7ad59394" (UID: "661cbfc6-4689-4825-8aed-fc9a7ad59394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:59.285391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.285242    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:59.285391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.285269    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:59.285391 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.285313    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661cbfc6-4689-4825-8aed-fc9a7ad59394-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:59.287093 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.287068    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-envoy" pod "661cbfc6-4689-4825-8aed-fc9a7ad59394" (UID: "661cbfc6-4689-4825-8aed-fc9a7ad59394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:59.287202 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.287153    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-kube-api-access-tftjs" pod "661cbfc6-4689-4825-8aed-fc9a7ad59394" (UID: "661cbfc6-4689-4825-8aed-fc9a7ad59394"). InnerVolumeSpecName "kube-api-access-tftjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:59.287591 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.287568    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-token" pod "661cbfc6-4689-4825-8aed-fc9a7ad59394" (UID: "661cbfc6-4689-4825-8aed-fc9a7ad59394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:59.287690 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.287632    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-podinfo" pod "661cbfc6-4689-4825-8aed-fc9a7ad59394" (UID: "661cbfc6-4689-4825-8aed-fc9a7ad59394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:59:59.385994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.385911    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:59.385994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.385942    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:59.385994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.385956    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftjs\" (UniqueName: \"kubernetes.io/projected/661cbfc6-4689-4825-8aed-fc9a7ad59394-kube-api-access-tftjs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:59.385994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.385969    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:59.385994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.385983    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:59.385994 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.385997    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661cbfc6-4689-4825-8aed-fc9a7ad59394-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 01:59:59.449226 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.449198    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"]
Nov 02 01:59:59.451938 ip-10-0-140-23 kubenswrapper[4851]: W1102 01:59:59.451912    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c418ec3_c826_465a_bc5a_d73a9043ec8e.slice/crio-a51e2afdc4325ed3ed293f3bc77114a7c74affe25abd18fc04e060daf1baca5a WatchSource:0}: Error finding container a51e2afdc4325ed3ed293f3bc77114a7c74affe25abd18fc04e060daf1baca5a: Status 404 returned error can't find the container with id a51e2afdc4325ed3ed293f3bc77114a7c74affe25abd18fc04e060daf1baca5a
Nov 02 01:59:59.602810 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.602772    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp" event={"ID":"2c418ec3-c826-465a-bc5a-d73a9043ec8e","Type":"ContainerStarted","Data":"4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84"}
Nov 02 01:59:59.603005 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.602821    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp" event={"ID":"2c418ec3-c826-465a-bc5a-d73a9043ec8e","Type":"ContainerStarted","Data":"a51e2afdc4325ed3ed293f3bc77114a7c74affe25abd18fc04e060daf1baca5a"}
Nov 02 01:59:59.604488 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.604462    4851 generic.go:356] "Generic (PLEG): container finished" podID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerID="c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe" exitCode=0
Nov 02 01:59:59.604604 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.604502    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"
Nov 02 01:59:59.604604 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.604519    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" event={"ID":"661cbfc6-4689-4825-8aed-fc9a7ad59394","Type":"ContainerDied","Data":"c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe"}
Nov 02 01:59:59.604604 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.604563    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg" event={"ID":"661cbfc6-4689-4825-8aed-fc9a7ad59394","Type":"ContainerDied","Data":"8549c59ba8c6a6f69488f9bab1af1c4ca9eecf03f40b5dfe78ba4ae0e283c6e4"}
Nov 02 01:59:59.604604 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.604587    4851 scope.go:122] "RemoveContainer" containerID="c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe"
Nov 02 01:59:59.614312 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.614271    4851 scope.go:122] "RemoveContainer" containerID="2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860"
Nov 02 01:59:59.621914 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.621872    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp" podStartSLOduration=1.621859144 podStartE2EDuration="1.621859144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:59.621139397 +0000 UTC m=+2704.926072224" watchObservedRunningTime="2026-11-02 01:59:59.621859144 +0000 UTC m=+2704.926791970"
Nov 02 01:59:59.623092 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.623071    4851 scope.go:122] "RemoveContainer" containerID="c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe"
Nov 02 01:59:59.623413 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:59:59.623390    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe\": container with ID starting with c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe not found: ID does not exist" containerID="c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe"
Nov 02 01:59:59.623483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.623425    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe"} err="failed to get container status \"c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe\": rpc error: code = NotFound desc = could not find container \"c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe\": container with ID starting with c41095cc9434fbe4ec1aacc279563cf14bd4a940a3636906bb6bb00d31c6dbbe not found: ID does not exist"
Nov 02 01:59:59.623483 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.623448    4851 scope.go:122] "RemoveContainer" containerID="2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860"
Nov 02 01:59:59.623715 ip-10-0-140-23 kubenswrapper[4851]: E1102 01:59:59.623698    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860\": container with ID starting with 2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860 not found: ID does not exist" containerID="2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860"
Nov 02 01:59:59.623765 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.623721    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860"} err="failed to get container status \"2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860\": rpc error: code = NotFound desc = could not find container \"2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860\": container with ID starting with 2fda9aaacbd1867292671156f7e26c896e5aeca49fb428071291832dafc1c860 not found: ID does not exist"
Nov 02 01:59:59.634801 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.634773    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"]
Nov 02 01:59:59.642212 ip-10-0-140-23 kubenswrapper[4851]: I1102 01:59:59.642147    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-693506-openshift-default-789b5887bffwwsg"]
Nov 02 02:00:00.134425 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.134396    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"]
Nov 02 02:00:00.134810 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.134797    4851 cpu_manager.go:414] "containerMap: removing container" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy"
Nov 02 02:00:00.134857 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.134811    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy"
Nov 02 02:00:00.134897 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.134889    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy"
Nov 02 02:00:00.134936 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.134902    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy"
Nov 02 02:00:00.139156 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.139138    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"
Nov 02 02:00:00.146699 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.146496    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:00:00.146699 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.146523    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:00:00.146699 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.146604    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 02:00:00.149669 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.149645    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"]
Nov 02 02:00:00.167600 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.167579    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 02:00:00.294033 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.293937    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc72q\" (UniqueName: \"kubernetes.io/projected/0141c062-a134-4c67-b8e3-4a824a320d41-kube-api-access-qc72q\") pod \"maas-api-key-cleanup-29893080-jtnxx\" (UID: \"0141c062-a134-4c67-b8e3-4a824a320d41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"
Nov 02 02:00:00.294208 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.294114    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0141c062-a134-4c67-b8e3-4a824a320d41-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893080-jtnxx\" (UID: \"0141c062-a134-4c67-b8e3-4a824a320d41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"
Nov 02 02:00:00.395417 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.395383    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0141c062-a134-4c67-b8e3-4a824a320d41-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893080-jtnxx\" (UID: \"0141c062-a134-4c67-b8e3-4a824a320d41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"
Nov 02 02:00:00.395600 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.395474    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc72q\" (UniqueName: \"kubernetes.io/projected/0141c062-a134-4c67-b8e3-4a824a320d41-kube-api-access-qc72q\") pod \"maas-api-key-cleanup-29893080-jtnxx\" (UID: \"0141c062-a134-4c67-b8e3-4a824a320d41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"
Nov 02 02:00:00.396212 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.396187    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0141c062-a134-4c67-b8e3-4a824a320d41-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893080-jtnxx\" (UID: \"0141c062-a134-4c67-b8e3-4a824a320d41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"
Nov 02 02:00:00.408062 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.408031    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc72q\" (UniqueName: \"kubernetes.io/projected/0141c062-a134-4c67-b8e3-4a824a320d41-kube-api-access-qc72q\") pod \"maas-api-key-cleanup-29893080-jtnxx\" (UID: \"0141c062-a134-4c67-b8e3-4a824a320d41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"
Nov 02 02:00:00.453649 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.453607    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"
Nov 02 02:00:00.734768 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.734726    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"]
Nov 02 02:00:00.737594 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:00:00.737567    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0141c062_a134_4c67_b8e3_4a824a320d41.slice/crio-c7e978b480ab55a770e2d9359ac1e338ebe8404eba83cbc0cddf62cceafe4445 WatchSource:0}: Error finding container c7e978b480ab55a770e2d9359ac1e338ebe8404eba83cbc0cddf62cceafe4445: Status 404 returned error can't find the container with id c7e978b480ab55a770e2d9359ac1e338ebe8404eba83cbc0cddf62cceafe4445
Nov 02 02:00:00.847556 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:00.847525    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" path="/var/lib/kubelet/pods/661cbfc6-4689-4825-8aed-fc9a7ad59394/volumes"
Nov 02 02:00:01.632542 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:01.632099    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" event={"ID":"0141c062-a134-4c67-b8e3-4a824a320d41","Type":"ContainerStarted","Data":"f4528aee8f60b8509bf517ca8d4079314ad326aecf63df4596baefd0cf291553"}
Nov 02 02:00:01.632542 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:01.632171    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" event={"ID":"0141c062-a134-4c67-b8e3-4a824a320d41","Type":"ContainerStarted","Data":"c7e978b480ab55a770e2d9359ac1e338ebe8404eba83cbc0cddf62cceafe4445"}
Nov 02 02:00:01.653235 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:01.653166    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" podStartSLOduration=1.653148499 podStartE2EDuration="1.653148499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:01.650650247 +0000 UTC m=+2706.955583074" watchObservedRunningTime="2026-11-02 02:00:01.653148499 +0000 UTC m=+2706.958081326"
Nov 02 02:00:02.657643 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:02.657604    4851 generic.go:356] "Generic (PLEG): container finished" podID="0141c062-a134-4c67-b8e3-4a824a320d41" containerID="f4528aee8f60b8509bf517ca8d4079314ad326aecf63df4596baefd0cf291553" exitCode=60
Nov 02 02:00:02.658495 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:02.658467    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" event={"ID":"0141c062-a134-4c67-b8e3-4a824a320d41","Type":"ContainerDied","Data":"f4528aee8f60b8509bf517ca8d4079314ad326aecf63df4596baefd0cf291553"}
Nov 02 02:00:02.659067 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:02.659046    4851 scope.go:122] "RemoveContainer" containerID="f4528aee8f60b8509bf517ca8d4079314ad326aecf63df4596baefd0cf291553"
Nov 02 02:00:03.562449 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:03.562408    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-44d86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:00:03.562793 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:03.562733    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-44d86" podUID="ee1b737b-620b-4615-b94f-0ae145b61646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:03.683429 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:03.683392    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" event={"ID":"0141c062-a134-4c67-b8e3-4a824a320d41","Type":"ContainerStarted","Data":"b2d05079178f56c75c775a5a975bf8da073da3cb4dbfd7469913ef967ad22373"}
Nov 02 02:00:03.727262 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:03.727224    4851 scope.go:122] "RemoveContainer" containerID="7ed5ac6edb3ac1dee4b5778f8c8692c8b0465820f549682bbe91c72984706fd9"
Nov 02 02:00:03.761583 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:03.761536    4851 scope.go:122] "RemoveContainer" containerID="aca90d98107e56287f2b0ed03cd88f982f16c5f80724653cc40942a34d55d841"
Nov 02 02:00:03.812358 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:03.811816    4851 scope.go:122] "RemoveContainer" containerID="15e8ce02d30812f1a2c38da845728a54d6e441113e3bf76f69f9c5e561284433"
Nov 02 02:00:03.823889 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:03.823862    4851 scope.go:122] "RemoveContainer" containerID="1e523df3a0b594e05a984d3c534847dec576be0f7150380671f581cca63a0609"
Nov 02 02:00:04.689973 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:04.689937    4851 generic.go:356] "Generic (PLEG): container finished" podID="0141c062-a134-4c67-b8e3-4a824a320d41" containerID="b2d05079178f56c75c775a5a975bf8da073da3cb4dbfd7469913ef967ad22373" exitCode=60
Nov 02 02:00:04.690458 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:04.689975    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" event={"ID":"0141c062-a134-4c67-b8e3-4a824a320d41","Type":"ContainerDied","Data":"b2d05079178f56c75c775a5a975bf8da073da3cb4dbfd7469913ef967ad22373"}
Nov 02 02:00:04.690458 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:04.690034    4851 scope.go:122] "RemoveContainer" containerID="f4528aee8f60b8509bf517ca8d4079314ad326aecf63df4596baefd0cf291553"
Nov 02 02:00:04.690458 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:04.690258    4851 scope.go:122] "RemoveContainer" containerID="b2d05079178f56c75c775a5a975bf8da073da3cb4dbfd7469913ef967ad22373"
Nov 02 02:00:04.690588 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:00:04.690561    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893080-jtnxx_odh-ai-gateway-infra(0141c062-a134-4c67-b8e3-4a824a320d41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" podUID="0141c062-a134-4c67-b8e3-4a824a320d41"
Nov 02 02:00:05.697708 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:05.697679    4851 scope.go:122] "RemoveContainer" containerID="b2d05079178f56c75c775a5a975bf8da073da3cb4dbfd7469913ef967ad22373"
Nov 02 02:00:05.698160 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:00:05.697999    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893080-jtnxx_odh-ai-gateway-infra(0141c062-a134-4c67-b8e3-4a824a320d41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" podUID="0141c062-a134-4c67-b8e3-4a824a320d41"
Nov 02 02:00:07.164472 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:07.164429    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb" podUID="a094bee7-3ba9-4b57-a857-995b218aad56" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:00:07.164901 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:07.164442    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87" podUID="c9b114d5-8090-481a-b8a5-f41b05effa8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:00:09.167134 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:09.167100    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 02:00:09.179569 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:09.179544    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 02:00:09.729378 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:09.729351    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 02:00:11.548124 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.548101    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 02:00:11.551182 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.551163    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 02:00:11.677866 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.677835    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a094bee7-3ba9-4b57-a857-995b218aad56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a094bee7-3ba9-4b57-a857-995b218aad56-plugins-config-volume\") pod \"a094bee7-3ba9-4b57-a857-995b218aad56\" (UID: \"a094bee7-3ba9-4b57-a857-995b218aad56\") "
Nov 02 02:00:11.678070 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.677948    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a094bee7-3ba9-4b57-a857-995b218aad56-kube-api-access-9mmgs\" (UniqueName: \"kubernetes.io/projected/a094bee7-3ba9-4b57-a857-995b218aad56-kube-api-access-9mmgs\") pod \"a094bee7-3ba9-4b57-a857-995b218aad56\" (UID: \"a094bee7-3ba9-4b57-a857-995b218aad56\") "
Nov 02 02:00:11.678070 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.677995    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b114d5-8090-481a-b8a5-f41b05effa8b-kube-api-access-l9kng\" (UniqueName: \"kubernetes.io/projected/c9b114d5-8090-481a-b8a5-f41b05effa8b-kube-api-access-l9kng\") pod \"c9b114d5-8090-481a-b8a5-f41b05effa8b\" (UID: \"c9b114d5-8090-481a-b8a5-f41b05effa8b\") "
Nov 02 02:00:11.678070 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.678020    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b114d5-8090-481a-b8a5-f41b05effa8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b114d5-8090-481a-b8a5-f41b05effa8b-plugins-config-volume\") pod \"c9b114d5-8090-481a-b8a5-f41b05effa8b\" (UID: \"c9b114d5-8090-481a-b8a5-f41b05effa8b\") "
Nov 02 02:00:11.678384 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.678357    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a094bee7-3ba9-4b57-a857-995b218aad56-plugins-config-volume" pod "a094bee7-3ba9-4b57-a857-995b218aad56" (UID: "a094bee7-3ba9-4b57-a857-995b218aad56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:11.678384 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.678375    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b114d5-8090-481a-b8a5-f41b05effa8b-plugins-config-volume" pod "c9b114d5-8090-481a-b8a5-f41b05effa8b" (UID: "c9b114d5-8090-481a-b8a5-f41b05effa8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:11.680467 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.680438    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a094bee7-3ba9-4b57-a857-995b218aad56-kube-api-access-9mmgs" pod "a094bee7-3ba9-4b57-a857-995b218aad56" (UID: "a094bee7-3ba9-4b57-a857-995b218aad56"). InnerVolumeSpecName "kube-api-access-9mmgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:11.680566 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.680492    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b114d5-8090-481a-b8a5-f41b05effa8b-kube-api-access-l9kng" pod "c9b114d5-8090-481a-b8a5-f41b05effa8b" (UID: "c9b114d5-8090-481a-b8a5-f41b05effa8b"). InnerVolumeSpecName "kube-api-access-l9kng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:11.729462 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.729420    4851 generic.go:356] "Generic (PLEG): container finished" podID="a094bee7-3ba9-4b57-a857-995b218aad56" containerID="ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b" exitCode=0
Nov 02 02:00:11.729675 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.729475    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"
Nov 02 02:00:11.729675 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.729486    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb" event={"ID":"a094bee7-3ba9-4b57-a857-995b218aad56","Type":"ContainerDied","Data":"ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b"}
Nov 02 02:00:11.729675 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.729544    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb" event={"ID":"a094bee7-3ba9-4b57-a857-995b218aad56","Type":"ContainerDied","Data":"3b6bdc030dcc8ad88edb23c72d661754baf009cc3fdf477da405e689119e7ecd"}
Nov 02 02:00:11.729675 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.729574    4851 scope.go:122] "RemoveContainer" containerID="ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b"
Nov 02 02:00:11.731050 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.731030    4851 generic.go:356] "Generic (PLEG): container finished" podID="c9b114d5-8090-481a-b8a5-f41b05effa8b" containerID="691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603" exitCode=0
Nov 02 02:00:11.731190 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.731069    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87" event={"ID":"c9b114d5-8090-481a-b8a5-f41b05effa8b","Type":"ContainerDied","Data":"691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603"}
Nov 02 02:00:11.731190 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.731079    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"
Nov 02 02:00:11.731190 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.731096    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87" event={"ID":"c9b114d5-8090-481a-b8a5-f41b05effa8b","Type":"ContainerDied","Data":"e9be3154a637ef1ba0b73df5385f4315a99dd774f18bfb70c092c5e13d879ec0"}
Nov 02 02:00:11.748104 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.742707    4851 scope.go:122] "RemoveContainer" containerID="ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b"
Nov 02 02:00:11.748104 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:00:11.743205    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b\": container with ID starting with ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b not found: ID does not exist" containerID="ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b"
Nov 02 02:00:11.748104 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.743238    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b"} err="failed to get container status \"ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b\": rpc error: code = NotFound desc = could not find container \"ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b\": container with ID starting with ddeccc4ad49c9cd350a933d620e2f5cf812fa757caa7a0cfb448d8e77800eb7b not found: ID does not exist"
Nov 02 02:00:11.748104 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.743265    4851 scope.go:122] "RemoveContainer" containerID="691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603"
Nov 02 02:00:11.755340 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.755316    4851 scope.go:122] "RemoveContainer" containerID="691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603"
Nov 02 02:00:11.755999 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:00:11.755964    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603\": container with ID starting with 691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603 not found: ID does not exist" containerID="691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603"
Nov 02 02:00:11.756147 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.756001    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603"} err="failed to get container status \"691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603\": rpc error: code = NotFound desc = could not find container \"691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603\": container with ID starting with 691dc19c7a7a36a97d6d52bd8b4a2a7afae423ab3763989a1515022392089603 not found: ID does not exist"
Nov 02 02:00:11.758355 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.758328    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"]
Nov 02 02:00:11.762551 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.762459    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-693506-68fd4cb94xsdjb"]
Nov 02 02:00:11.773275 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.773212    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"]
Nov 02 02:00:11.777442 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.777417    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-693506-77f7ch4m87"]
Nov 02 02:00:11.779020 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.778999    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mmgs\" (UniqueName: \"kubernetes.io/projected/a094bee7-3ba9-4b57-a857-995b218aad56-kube-api-access-9mmgs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:11.779123 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.779022    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9kng\" (UniqueName: \"kubernetes.io/projected/c9b114d5-8090-481a-b8a5-f41b05effa8b-kube-api-access-l9kng\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:11.779123 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.779034    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b114d5-8090-481a-b8a5-f41b05effa8b-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:11.779123 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:11.779043    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a094bee7-3ba9-4b57-a857-995b218aad56-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:12.845763 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:12.845725    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a094bee7-3ba9-4b57-a857-995b218aad56" path="/var/lib/kubelet/pods/a094bee7-3ba9-4b57-a857-995b218aad56/volumes"
Nov 02 02:00:12.846342 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:12.846313    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b114d5-8090-481a-b8a5-f41b05effa8b" path="/var/lib/kubelet/pods/c9b114d5-8090-481a-b8a5-f41b05effa8b/volumes"
Nov 02 02:00:14.261244 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.261206    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"]
Nov 02 02:00:14.261699 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.261606    4851 cpu_manager.go:414] "containerMap: removing container" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy"
Nov 02 02:00:14.261699 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.261618    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661cbfc6-4689-4825-8aed-fc9a7ad59394" containerName="istio-proxy"
Nov 02 02:00:14.261699 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.261635    4851 cpu_manager.go:414] "containerMap: removing container" podUID="a094bee7-3ba9-4b57-a857-995b218aad56" containerName="payload-processing"
Nov 02 02:00:14.261699 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.261640    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a094bee7-3ba9-4b57-a857-995b218aad56" containerName="payload-processing"
Nov 02 02:00:14.261699 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.261660    4851 cpu_manager.go:414] "containerMap: removing container" podUID="c9b114d5-8090-481a-b8a5-f41b05effa8b" containerName="payload-pre-processing"
Nov 02 02:00:14.261699 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.261667    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b114d5-8090-481a-b8a5-f41b05effa8b" containerName="payload-pre-processing"
Nov 02 02:00:14.261935 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.261741    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="a094bee7-3ba9-4b57-a857-995b218aad56" containerName="payload-processing"
Nov 02 02:00:14.261935 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.261752    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b114d5-8090-481a-b8a5-f41b05effa8b" containerName="payload-pre-processing"
Nov 02 02:00:14.266219 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.266199    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.272412 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.272388    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-452a45db-kserve-self-signed-certs\""
Nov 02 02:00:14.318687 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.318658    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"]
Nov 02 02:00:14.406068 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.406031    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-dshm\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.406480 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.406454    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cgdz\" (UniqueName: \"kubernetes.io/projected/8125b06a-f1eb-49de-80ac-61a79d4790c0-kube-api-access-9cgdz\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.406675 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.406661    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8125b06a-f1eb-49de-80ac-61a79d4790c0-tls-certs\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.406898 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.406881    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-home\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.407111 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.407093    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-model-cache\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.407215 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.407126    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-tmp-dir\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.507640 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.507607    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cgdz\" (UniqueName: \"kubernetes.io/projected/8125b06a-f1eb-49de-80ac-61a79d4790c0-kube-api-access-9cgdz\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.507873 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.507654    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8125b06a-f1eb-49de-80ac-61a79d4790c0-tls-certs\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.507873 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.507736    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-home\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.507873 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.507770    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-model-cache\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.507873 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.507794    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-tmp-dir\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.507873 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.507846    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-dshm\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.508214 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.508183    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-home\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.508386 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.508220    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-model-cache\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.508386 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.508247    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-tmp-dir\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.510356 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.510332    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-dshm\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.510551 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.510533    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8125b06a-f1eb-49de-80ac-61a79d4790c0-tls-certs\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.519860 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.519810    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cgdz\" (UniqueName: \"kubernetes.io/projected/8125b06a-f1eb-49de-80ac-61a79d4790c0-kube-api-access-9cgdz\") pod \"e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") " pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.578895 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.578862    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:14.857268 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:14.857241    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"]
Nov 02 02:00:14.860011 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:00:14.859978    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8125b06a_f1eb_49de_80ac_61a79d4790c0.slice/crio-b518da88dd4a69db43a6ddedce78a4cdcaa26066385e035f4b5c800567ab8e8c WatchSource:0}: Error finding container b518da88dd4a69db43a6ddedce78a4cdcaa26066385e035f4b5c800567ab8e8c: Status 404 returned error can't find the container with id b518da88dd4a69db43a6ddedce78a4cdcaa26066385e035f4b5c800567ab8e8c
Nov 02 02:00:15.754359 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:15.754320    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf" event={"ID":"8125b06a-f1eb-49de-80ac-61a79d4790c0","Type":"ContainerStarted","Data":"06c95c3c651ef50fa4b7e8b71aa31a491d9980e81a8e7ef588b8fe2087b38a5e"}
Nov 02 02:00:15.754359 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:15.754361    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf" event={"ID":"8125b06a-f1eb-49de-80ac-61a79d4790c0","Type":"ContainerStarted","Data":"b518da88dd4a69db43a6ddedce78a4cdcaa26066385e035f4b5c800567ab8e8c"}
Nov 02 02:00:15.774461 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:15.774402    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf" podStartSLOduration=1.774388217 podStartE2EDuration="1.774388217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:15.772671725 +0000 UTC m=+2721.077604563" watchObservedRunningTime="2026-11-02 02:00:15.774388217 +0000 UTC m=+2721.079321042"
Nov 02 02:00:15.838974 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:15.838943    4851 scope.go:122] "RemoveContainer" containerID="b2d05079178f56c75c775a5a975bf8da073da3cb4dbfd7469913ef967ad22373"
Nov 02 02:00:16.579433 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:16.579387    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:16.765716 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:16.765669    4851 generic.go:356] "Generic (PLEG): container finished" podID="0141c062-a134-4c67-b8e3-4a824a320d41" containerID="c891cdf21afcdb3d63107a71359961e3ac42b52a6b63f6896351fa263bff26af" exitCode=60
Nov 02 02:00:16.766238 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:16.765736    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" event={"ID":"0141c062-a134-4c67-b8e3-4a824a320d41","Type":"ContainerDied","Data":"c891cdf21afcdb3d63107a71359961e3ac42b52a6b63f6896351fa263bff26af"}
Nov 02 02:00:16.766238 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:16.765852    4851 scope.go:122] "RemoveContainer" containerID="b2d05079178f56c75c775a5a975bf8da073da3cb4dbfd7469913ef967ad22373"
Nov 02 02:00:16.766238 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:16.766237    4851 scope.go:122] "RemoveContainer" containerID="c891cdf21afcdb3d63107a71359961e3ac42b52a6b63f6896351fa263bff26af"
Nov 02 02:00:16.766548 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:00:16.766524    4851 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893080-jtnxx_odh-ai-gateway-infra(0141c062-a134-4c67-b8e3-4a824a320d41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" podUID="0141c062-a134-4c67-b8e3-4a824a320d41"
Nov 02 02:00:17.669667 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:00:17.669620    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:00:17.801317 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:17.796829    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"]
Nov 02 02:00:18.490529 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:18.490497    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"
Nov 02 02:00:18.673394 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:18.659821    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0141c062-a134-4c67-b8e3-4a824a320d41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0141c062-a134-4c67-b8e3-4a824a320d41-maas-api-service-ca\") pod \"0141c062-a134-4c67-b8e3-4a824a320d41\" (UID: \"0141c062-a134-4c67-b8e3-4a824a320d41\") "
Nov 02 02:00:18.693544 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:18.693501    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0141c062-a134-4c67-b8e3-4a824a320d41-kube-api-access-qc72q\" (UniqueName: \"kubernetes.io/projected/0141c062-a134-4c67-b8e3-4a824a320d41-kube-api-access-qc72q\") pod \"0141c062-a134-4c67-b8e3-4a824a320d41\" (UID: \"0141c062-a134-4c67-b8e3-4a824a320d41\") "
Nov 02 02:00:18.722741 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:18.677551    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0141c062-a134-4c67-b8e3-4a824a320d41-maas-api-service-ca" pod "0141c062-a134-4c67-b8e3-4a824a320d41" (UID: "0141c062-a134-4c67-b8e3-4a824a320d41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:18.767321 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:18.765484    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0141c062-a134-4c67-b8e3-4a824a320d41-kube-api-access-qc72q" pod "0141c062-a134-4c67-b8e3-4a824a320d41" (UID: "0141c062-a134-4c67-b8e3-4a824a320d41"). InnerVolumeSpecName "kube-api-access-qc72q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:18.831074 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:18.827928    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc72q\" (UniqueName: \"kubernetes.io/projected/0141c062-a134-4c67-b8e3-4a824a320d41-kube-api-access-qc72q\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:18.831074 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:18.827970    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0141c062-a134-4c67-b8e3-4a824a320d41-maas-api-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:19.018062 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:19.017891    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx" event={"ID":"0141c062-a134-4c67-b8e3-4a824a320d41","Type":"ContainerDied","Data":"c7e978b480ab55a770e2d9359ac1e338ebe8404eba83cbc0cddf62cceafe4445"}
Nov 02 02:00:19.018062 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:19.017974    4851 scope.go:122] "RemoveContainer" containerID="c891cdf21afcdb3d63107a71359961e3ac42b52a6b63f6896351fa263bff26af"
Nov 02 02:00:19.018436 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:19.018130    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"
Nov 02 02:00:19.071325 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:19.069345    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"]
Nov 02 02:00:19.079256 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:19.079208    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-jtnxx"]
Nov 02 02:00:20.845348 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:20.845282    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0141c062-a134-4c67-b8e3-4a824a320d41" path="/var/lib/kubelet/pods/0141c062-a134-4c67-b8e3-4a824a320d41/volumes"
Nov 02 02:00:24.579260 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:24.579220    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:24.591912 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:24.591888    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:25.066466 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:25.066430    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:00:31.521138 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.521100    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"]
Nov 02 02:00:31.521625 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.521530    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0141c062-a134-4c67-b8e3-4a824a320d41" containerName="cleanup"
Nov 02 02:00:31.521625 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.521543    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0141c062-a134-4c67-b8e3-4a824a320d41" containerName="cleanup"
Nov 02 02:00:31.521625 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.521554    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0141c062-a134-4c67-b8e3-4a824a320d41" containerName="cleanup"
Nov 02 02:00:31.521625 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.521559    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0141c062-a134-4c67-b8e3-4a824a320d41" containerName="cleanup"
Nov 02 02:00:31.521796 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.521640    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0141c062-a134-4c67-b8e3-4a824a320d41" containerName="cleanup"
Nov 02 02:00:31.521796 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.521652    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0141c062-a134-4c67-b8e3-4a824a320d41" containerName="cleanup"
Nov 02 02:00:31.521796 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.521657    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="0141c062-a134-4c67-b8e3-4a824a320d41" containerName="cleanup"
Nov 02 02:00:31.524535 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.524520    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:31.532193 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.532164    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:00:31.532371 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.532238    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:00:31.532371 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.532319    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:00:31.532505 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.532455    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 02:00:31.533639 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.533614    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"]
Nov 02 02:00:31.686622 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.686574    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xxgsr\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:31.686874 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.686644    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xxgsr\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:31.686874 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.686777    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j99rr\" (UniqueName: \"kubernetes.io/projected/91151a8c-d754-48b3-a47f-2dc07c065fb4-kube-api-access-j99rr\") pod \"maas-api-revoke-sub-452b6d7b-xxgsr\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:31.787819 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.787721    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xxgsr\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:31.787819 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.787808    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99rr\" (UniqueName: \"kubernetes.io/projected/91151a8c-d754-48b3-a47f-2dc07c065fb4-kube-api-access-j99rr\") pod \"maas-api-revoke-sub-452b6d7b-xxgsr\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:31.788047 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.787861    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xxgsr\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:31.788486 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.788467    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xxgsr\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:31.790334 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.790317    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xxgsr\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:31.800096 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.800077    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99rr\" (UniqueName: \"kubernetes.io/projected/91151a8c-d754-48b3-a47f-2dc07c065fb4-kube-api-access-j99rr\") pod \"maas-api-revoke-sub-452b6d7b-xxgsr\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:31.839382 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:31.839345    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:32.132722 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:32.132693    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"]
Nov 02 02:00:32.135398 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:00:32.135368    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91151a8c_d754_48b3_a47f_2dc07c065fb4.slice/crio-a849e9cb16fdb2a07c2902a08b87ed19d5998778b69dcf7f4de9a499f8e5fa89 WatchSource:0}: Error finding container a849e9cb16fdb2a07c2902a08b87ed19d5998778b69dcf7f4de9a499f8e5fa89: Status 404 returned error can't find the container with id a849e9cb16fdb2a07c2902a08b87ed19d5998778b69dcf7f4de9a499f8e5fa89
Nov 02 02:00:33.098891 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:33.098702    4851 generic.go:356] "Generic (PLEG): container finished" podID="91151a8c-d754-48b3-a47f-2dc07c065fb4" containerID="b1ab088c71f99366d75e28a3ee4459fdeb8e5c27f40bf922af039a3b40760486" exitCode=0
Nov 02 02:00:33.098891 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:33.098798    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr" event={"ID":"91151a8c-d754-48b3-a47f-2dc07c065fb4","Type":"ContainerDied","Data":"b1ab088c71f99366d75e28a3ee4459fdeb8e5c27f40bf922af039a3b40760486"}
Nov 02 02:00:33.098891 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:33.098851    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr" event={"ID":"91151a8c-d754-48b3-a47f-2dc07c065fb4","Type":"ContainerStarted","Data":"a849e9cb16fdb2a07c2902a08b87ed19d5998778b69dcf7f4de9a499f8e5fa89"}
Nov 02 02:00:35.125513 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.123505    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:35.162383 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.162143    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr" event={"ID":"91151a8c-d754-48b3-a47f-2dc07c065fb4","Type":"ContainerDied","Data":"a849e9cb16fdb2a07c2902a08b87ed19d5998778b69dcf7f4de9a499f8e5fa89"}
Nov 02 02:00:35.162383 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.162222    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a849e9cb16fdb2a07c2902a08b87ed19d5998778b69dcf7f4de9a499f8e5fa89"
Nov 02 02:00:35.162383 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.162324    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"
Nov 02 02:00:35.170606 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.170570    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-xks-ca\") pod \"91151a8c-d754-48b3-a47f-2dc07c065fb4\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") "
Nov 02 02:00:35.171721 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.170909    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-openshift-ca\") pod \"91151a8c-d754-48b3-a47f-2dc07c065fb4\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") "
Nov 02 02:00:35.171721 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.171049    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91151a8c-d754-48b3-a47f-2dc07c065fb4-kube-api-access-j99rr\" (UniqueName: \"kubernetes.io/projected/91151a8c-d754-48b3-a47f-2dc07c065fb4-kube-api-access-j99rr\") pod \"91151a8c-d754-48b3-a47f-2dc07c065fb4\" (UID: \"91151a8c-d754-48b3-a47f-2dc07c065fb4\") "
Nov 02 02:00:35.179581 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.179526    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-openshift-ca" pod "91151a8c-d754-48b3-a47f-2dc07c065fb4" (UID: "91151a8c-d754-48b3-a47f-2dc07c065fb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:35.198316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.186807    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-xks-ca" pod "91151a8c-d754-48b3-a47f-2dc07c065fb4" (UID: "91151a8c-d754-48b3-a47f-2dc07c065fb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:00:35.199312 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.199262    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91151a8c-d754-48b3-a47f-2dc07c065fb4-kube-api-access-j99rr" pod "91151a8c-d754-48b3-a47f-2dc07c065fb4" (UID: "91151a8c-d754-48b3-a47f-2dc07c065fb4"). InnerVolumeSpecName "kube-api-access-j99rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:35.293421 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.293364    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j99rr\" (UniqueName: \"kubernetes.io/projected/91151a8c-d754-48b3-a47f-2dc07c065fb4-kube-api-access-j99rr\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.369443 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.369402    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.399338 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.395448    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91151a8c-d754-48b3-a47f-2dc07c065fb4-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.752314 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.751435    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dvgpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:00:35.752314 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.751548    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dvgpc" podUID="2aa0e67e-2315-4bba-8060-e7d7b101918c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:35.776317 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.775393    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kw5w7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:00:35.776317 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:35.775454    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" podUID="cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:42.360618 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:42.360577    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"]
Nov 02 02:00:44.651727 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:44.651688    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"]
Nov 02 02:00:47.661884 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:00:47.661847    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:00:48.290691 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.290654    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"]
Nov 02 02:00:48.291268 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.291246    4851 cpu_manager.go:414] "containerMap: removing container" podUID="0141c062-a134-4c67-b8e3-4a824a320d41" containerName="cleanup"
Nov 02 02:00:48.291420 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.291272    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0141c062-a134-4c67-b8e3-4a824a320d41" containerName="cleanup"
Nov 02 02:00:48.291420 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.291323    4851 cpu_manager.go:414] "containerMap: removing container" podUID="91151a8c-d754-48b3-a47f-2dc07c065fb4" containerName="revoke-keys"
Nov 02 02:00:48.291420 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.291332    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91151a8c-d754-48b3-a47f-2dc07c065fb4" containerName="revoke-keys"
Nov 02 02:00:48.291612 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.291486    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="91151a8c-d754-48b3-a47f-2dc07c065fb4" containerName="revoke-keys"
Nov 02 02:00:48.295170 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.295138    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.302576 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.302548    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 02:00:48.303878 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.303854    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:00:48.307471 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.307445    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"]
Nov 02 02:00:48.308361 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.308180    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:00:48.308522 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.308244    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:00:48.379447 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.379388    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wks9c\" (UniqueName: \"kubernetes.io/projected/5fc2408c-0e37-4e02-989a-3682209fadf6-kube-api-access-wks9c\") pod \"maas-api-revoke-sub-3e31a506-z5zq5\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.379682 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.379459    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-z5zq5\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.379763 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.379677    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-z5zq5\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.480682 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.480624    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-z5zq5\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.480920 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.480716    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wks9c\" (UniqueName: \"kubernetes.io/projected/5fc2408c-0e37-4e02-989a-3682209fadf6-kube-api-access-wks9c\") pod \"maas-api-revoke-sub-3e31a506-z5zq5\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.480920 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.480877    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-z5zq5\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.481811 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.481775    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-z5zq5\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.484032 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.483994    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-z5zq5\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.494348 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.494309    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wks9c\" (UniqueName: \"kubernetes.io/projected/5fc2408c-0e37-4e02-989a-3682209fadf6-kube-api-access-wks9c\") pod \"maas-api-revoke-sub-3e31a506-z5zq5\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.631558 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.631453    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:48.955159 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:48.940828    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"]
Nov 02 02:00:48.998777 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:00:48.995499    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc2408c_0e37_4e02_989a_3682209fadf6.slice/crio-c41caed39dd3b94d44310ef4dbed2ddfe6a3c9c866aeb59a5df88da100389ea5 WatchSource:0}: Error finding container c41caed39dd3b94d44310ef4dbed2ddfe6a3c9c866aeb59a5df88da100389ea5: Status 404 returned error can't find the container with id c41caed39dd3b94d44310ef4dbed2ddfe6a3c9c866aeb59a5df88da100389ea5
Nov 02 02:00:49.290162 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:49.289991    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5" event={"ID":"5fc2408c-0e37-4e02-989a-3682209fadf6","Type":"ContainerStarted","Data":"c41caed39dd3b94d44310ef4dbed2ddfe6a3c9c866aeb59a5df88da100389ea5"}
Nov 02 02:00:50.358355 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:50.353401    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5" event={"ID":"5fc2408c-0e37-4e02-989a-3682209fadf6","Type":"ContainerStarted","Data":"10037cea532b32005b0628614146d61d7ffa9de16c04b981fffec6e02776c319"}
Nov 02 02:00:50.402967 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:50.402896    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5" podStartSLOduration=2.402873353 podStartE2EDuration="2.402873353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:50.399628964 +0000 UTC m=+2755.704561806" watchObservedRunningTime="2026-11-02 02:00:50.402873353 +0000 UTC m=+2755.707806181"
Nov 02 02:00:50.766099 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:50.765863    4851 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kw5w7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:00:50.766099 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:50.765990    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kw5w7" podUID="cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:51.360186 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:51.360148    4851 generic.go:356] "Generic (PLEG): container finished" podID="5fc2408c-0e37-4e02-989a-3682209fadf6" containerID="10037cea532b32005b0628614146d61d7ffa9de16c04b981fffec6e02776c319" exitCode=0
Nov 02 02:00:51.360631 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:51.360191    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5" event={"ID":"5fc2408c-0e37-4e02-989a-3682209fadf6","Type":"ContainerDied","Data":"10037cea532b32005b0628614146d61d7ffa9de16c04b981fffec6e02776c319"}
Nov 02 02:00:52.506632 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:52.506608    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:52.577643 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:52.577612    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc2408c-0e37-4e02-989a-3682209fadf6-kube-api-access-wks9c\" (UniqueName: \"kubernetes.io/projected/5fc2408c-0e37-4e02-989a-3682209fadf6-kube-api-access-wks9c\") pod \"5fc2408c-0e37-4e02-989a-3682209fadf6\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") "
Nov 02 02:00:52.577837 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:52.577731    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-openshift-ca\") pod \"5fc2408c-0e37-4e02-989a-3682209fadf6\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") "
Nov 02 02:00:52.577837 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:52.577771    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-xks-ca\") pod \"5fc2408c-0e37-4e02-989a-3682209fadf6\" (UID: \"5fc2408c-0e37-4e02-989a-3682209fadf6\") "
Nov 02 02:00:52.578153 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:52.578125    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-openshift-ca" pod "5fc2408c-0e37-4e02-989a-3682209fadf6" (UID: "5fc2408c-0e37-4e02-989a-3682209fadf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:52.579898 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:52.579872    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc2408c-0e37-4e02-989a-3682209fadf6-kube-api-access-wks9c" pod "5fc2408c-0e37-4e02-989a-3682209fadf6" (UID: "5fc2408c-0e37-4e02-989a-3682209fadf6"). InnerVolumeSpecName "kube-api-access-wks9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:52.580021 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:52.579948    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-xks-ca" pod "5fc2408c-0e37-4e02-989a-3682209fadf6" (UID: "5fc2408c-0e37-4e02-989a-3682209fadf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:00:52.679470 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:52.679416    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wks9c\" (UniqueName: \"kubernetes.io/projected/5fc2408c-0e37-4e02-989a-3682209fadf6-kube-api-access-wks9c\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:52.679470 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:52.679466    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:52.679470 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:52.679481    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc2408c-0e37-4e02-989a-3682209fadf6-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:53.371773 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:53.371743    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"
Nov 02 02:00:53.371773 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:53.371759    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5" event={"ID":"5fc2408c-0e37-4e02-989a-3682209fadf6","Type":"ContainerDied","Data":"c41caed39dd3b94d44310ef4dbed2ddfe6a3c9c866aeb59a5df88da100389ea5"}
Nov 02 02:00:53.372004 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:53.371801    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c41caed39dd3b94d44310ef4dbed2ddfe6a3c9c866aeb59a5df88da100389ea5"
Nov 02 02:00:57.381985 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.381919    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp" podUID="2c418ec3-c826-465a-bc5a-d73a9043ec8e" containerName="main" containerID="cri-o://4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84" gracePeriod=45
Nov 02 02:00:57.696391 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.696351    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 02:00:57.833015 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.832981    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tmp-dir\") pod \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") "
Nov 02 02:00:57.833232 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833046    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-home\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-home\") pod \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") "
Nov 02 02:00:57.833232 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833075    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c418ec3-c826-465a-bc5a-d73a9043ec8e-kube-api-access-45n7p\" (UniqueName: \"kubernetes.io/projected/2c418ec3-c826-465a-bc5a-d73a9043ec8e-kube-api-access-45n7p\") pod \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") "
Nov 02 02:00:57.833232 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833093    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-model-cache\") pod \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") "
Nov 02 02:00:57.833232 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833137    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-dshm\") pod \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") "
Nov 02 02:00:57.833232 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833187    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tls-certs\") pod \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\" (UID: \"2c418ec3-c826-465a-bc5a-d73a9043ec8e\") "
Nov 02 02:00:57.833572 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833318    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tmp-dir" pod "2c418ec3-c826-465a-bc5a-d73a9043ec8e" (UID: "2c418ec3-c826-465a-bc5a-d73a9043ec8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:57.833572 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833405    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-model-cache" pod "2c418ec3-c826-465a-bc5a-d73a9043ec8e" (UID: "2c418ec3-c826-465a-bc5a-d73a9043ec8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:57.833572 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833423    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-home" pod "2c418ec3-c826-465a-bc5a-d73a9043ec8e" (UID: "2c418ec3-c826-465a-bc5a-d73a9043ec8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:57.833739 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833615    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:57.833739 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833636    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:57.833739 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.833677    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:57.835685 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.835651    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tls-certs" pod "2c418ec3-c826-465a-bc5a-d73a9043ec8e" (UID: "2c418ec3-c826-465a-bc5a-d73a9043ec8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:00:57.835685 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.835663    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-dshm" pod "2c418ec3-c826-465a-bc5a-d73a9043ec8e" (UID: "2c418ec3-c826-465a-bc5a-d73a9043ec8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:57.835977 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.835716    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c418ec3-c826-465a-bc5a-d73a9043ec8e-kube-api-access-45n7p" pod "2c418ec3-c826-465a-bc5a-d73a9043ec8e" (UID: "2c418ec3-c826-465a-bc5a-d73a9043ec8e"). InnerVolumeSpecName "kube-api-access-45n7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:57.934894 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.934858    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45n7p\" (UniqueName: \"kubernetes.io/projected/2c418ec3-c826-465a-bc5a-d73a9043ec8e-kube-api-access-45n7p\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:57.934894 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.934895    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c418ec3-c826-465a-bc5a-d73a9043ec8e-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:57.935146 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:57.934910    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c418ec3-c826-465a-bc5a-d73a9043ec8e-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:00:58.399062 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:58.398970    4851 generic.go:356] "Generic (PLEG): container finished" podID="2c418ec3-c826-465a-bc5a-d73a9043ec8e" containerID="4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84" exitCode=0
Nov 02 02:00:58.399062 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:58.399019    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"
Nov 02 02:00:58.399589 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:58.399062    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp" event={"ID":"2c418ec3-c826-465a-bc5a-d73a9043ec8e","Type":"ContainerDied","Data":"4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84"}
Nov 02 02:00:58.399589 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:58.399108    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp" event={"ID":"2c418ec3-c826-465a-bc5a-d73a9043ec8e","Type":"ContainerDied","Data":"a51e2afdc4325ed3ed293f3bc77114a7c74affe25abd18fc04e060daf1baca5a"}
Nov 02 02:00:58.399589 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:58.399131    4851 scope.go:122] "RemoveContainer" containerID="4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84"
Nov 02 02:00:58.409157 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:58.409135    4851 scope.go:122] "RemoveContainer" containerID="4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84"
Nov 02 02:00:58.409469 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:00:58.409448    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84\": container with ID starting with 4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84 not found: ID does not exist" containerID="4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84"
Nov 02 02:00:58.409547 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:58.409478    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84"} err="failed to get container status \"4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84\": rpc error: code = NotFound desc = could not find container \"4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84\": container with ID starting with 4aecab63a660d54c90366ead45d56da2272fccc57fdebe6181f31347839bea84 not found: ID does not exist"
Nov 02 02:00:58.421186 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:58.421156    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"]
Nov 02 02:00:58.425038 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:58.425016    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-452a45db-kserve-9fcb84f8f-f6wvp"]
Nov 02 02:00:58.844325 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:58.844275    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c418ec3-c826-465a-bc5a-d73a9043ec8e" path="/var/lib/kubelet/pods/2c418ec3-c826-465a-bc5a-d73a9043ec8e/volumes"
Nov 02 02:00:59.701555 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:00:59.697477    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf" podUID="8125b06a-f1eb-49de-80ac-61a79d4790c0" containerName="main" containerID="cri-o://06c95c3c651ef50fa4b7e8b71aa31a491d9980e81a8e7ef588b8fe2087b38a5e" gracePeriod=45
Nov 02 02:01:00.060867 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.060761    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf" podUID="8125b06a-f1eb-49de-80ac-61a79d4790c0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.56:8000/ready\": dial tcp 10.132.1.56:8000: connect: connection refused"
Nov 02 02:01:00.563753 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.563717    4851 generic.go:356] "Generic (PLEG): container finished" podID="8125b06a-f1eb-49de-80ac-61a79d4790c0" containerID="06c95c3c651ef50fa4b7e8b71aa31a491d9980e81a8e7ef588b8fe2087b38a5e" exitCode=0
Nov 02 02:01:00.564054 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.563982    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf" event={"ID":"8125b06a-f1eb-49de-80ac-61a79d4790c0","Type":"ContainerDied","Data":"06c95c3c651ef50fa4b7e8b71aa31a491d9980e81a8e7ef588b8fe2087b38a5e"}
Nov 02 02:01:00.594602 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.594568    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:01:00.683316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.682615    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-model-cache\") pod \"8125b06a-f1eb-49de-80ac-61a79d4790c0\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") "
Nov 02 02:01:00.683316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.682746    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8125b06a-f1eb-49de-80ac-61a79d4790c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/8125b06a-f1eb-49de-80ac-61a79d4790c0-tls-certs\") pod \"8125b06a-f1eb-49de-80ac-61a79d4790c0\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") "
Nov 02 02:01:00.683316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.682833    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8125b06a-f1eb-49de-80ac-61a79d4790c0-kube-api-access-9cgdz\" (UniqueName: \"kubernetes.io/projected/8125b06a-f1eb-49de-80ac-61a79d4790c0-kube-api-access-9cgdz\") pod \"8125b06a-f1eb-49de-80ac-61a79d4790c0\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") "
Nov 02 02:01:00.683316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.682889    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-dshm\") pod \"8125b06a-f1eb-49de-80ac-61a79d4790c0\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") "
Nov 02 02:01:00.683316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.682960    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-home\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-home\") pod \"8125b06a-f1eb-49de-80ac-61a79d4790c0\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") "
Nov 02 02:01:00.683316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.682992    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-tmp-dir\") pod \"8125b06a-f1eb-49de-80ac-61a79d4790c0\" (UID: \"8125b06a-f1eb-49de-80ac-61a79d4790c0\") "
Nov 02 02:01:00.709370 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.707407    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-tmp-dir" pod "8125b06a-f1eb-49de-80ac-61a79d4790c0" (UID: "8125b06a-f1eb-49de-80ac-61a79d4790c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:00.710137 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.710107    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-home" pod "8125b06a-f1eb-49de-80ac-61a79d4790c0" (UID: "8125b06a-f1eb-49de-80ac-61a79d4790c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:00.710534 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.710514    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-model-cache" pod "8125b06a-f1eb-49de-80ac-61a79d4790c0" (UID: "8125b06a-f1eb-49de-80ac-61a79d4790c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:00.751590 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.751560    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8125b06a-f1eb-49de-80ac-61a79d4790c0-kube-api-access-9cgdz" pod "8125b06a-f1eb-49de-80ac-61a79d4790c0" (UID: "8125b06a-f1eb-49de-80ac-61a79d4790c0"). InnerVolumeSpecName "kube-api-access-9cgdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:00.752901 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.752872    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-dshm" pod "8125b06a-f1eb-49de-80ac-61a79d4790c0" (UID: "8125b06a-f1eb-49de-80ac-61a79d4790c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:00.765892 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.765858    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8125b06a-f1eb-49de-80ac-61a79d4790c0-tls-certs" pod "8125b06a-f1eb-49de-80ac-61a79d4790c0" (UID: "8125b06a-f1eb-49de-80ac-61a79d4790c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:00.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.786904    4851 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-tmp-dir\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:00.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.786961    4851 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-model-cache\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:00.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.786978    4851 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8125b06a-f1eb-49de-80ac-61a79d4790c0-tls-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:00.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.786994    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cgdz\" (UniqueName: \"kubernetes.io/projected/8125b06a-f1eb-49de-80ac-61a79d4790c0-kube-api-access-9cgdz\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:00.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.787010    4851 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-dshm\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:00.787050 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:00.787023    4851 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8125b06a-f1eb-49de-80ac-61a79d4790c0-home\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:01.579317 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:01.578986    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf" event={"ID":"8125b06a-f1eb-49de-80ac-61a79d4790c0","Type":"ContainerDied","Data":"b518da88dd4a69db43a6ddedce78a4cdcaa26066385e035f4b5c800567ab8e8c"}
Nov 02 02:01:01.579317 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:01.579046    4851 scope.go:122] "RemoveContainer" containerID="06c95c3c651ef50fa4b7e8b71aa31a491d9980e81a8e7ef588b8fe2087b38a5e"
Nov 02 02:01:01.579317 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:01.579158    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"
Nov 02 02:01:01.629053 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:01.621813    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"]
Nov 02 02:01:01.629053 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:01.625631    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-452a45db-kserve-657dd86b85-r5lzf"]
Nov 02 02:01:02.868449 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:02.868414    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8125b06a-f1eb-49de-80ac-61a79d4790c0" path="/var/lib/kubelet/pods/8125b06a-f1eb-49de-80ac-61a79d4790c0/volumes"
Nov 02 02:01:03.928947 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:03.928910    4851 scope.go:122] "RemoveContainer" containerID="75fb92a582c68d72f9caa37a7bd6eed9b46e580e636edeba4c6d4cbee4eda9ed"
Nov 02 02:01:03.968361 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:03.966469    4851 scope.go:122] "RemoveContainer" containerID="339123ba796fcd548e584cc6e23d4db8b432f9d806ee8a3ad090f48b9eb9d4b0"
Nov 02 02:01:04.024222 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:04.024188    4851 scope.go:122] "RemoveContainer" containerID="dda65216e5dfe175e421b17f3071d3423deac0e8a71d5cd49aa4bbb8823b7097"
Nov 02 02:01:08.550066 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.550029    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"]
Nov 02 02:01:08.550594 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.550430    4851 cpu_manager.go:414] "containerMap: removing container" podUID="2c418ec3-c826-465a-bc5a-d73a9043ec8e" containerName="main"
Nov 02 02:01:08.550594 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.550441    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c418ec3-c826-465a-bc5a-d73a9043ec8e" containerName="main"
Nov 02 02:01:08.550594 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.550456    4851 cpu_manager.go:414] "containerMap: removing container" podUID="8125b06a-f1eb-49de-80ac-61a79d4790c0" containerName="main"
Nov 02 02:01:08.550594 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.550461    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8125b06a-f1eb-49de-80ac-61a79d4790c0" containerName="main"
Nov 02 02:01:08.550594 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.550475    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5fc2408c-0e37-4e02-989a-3682209fadf6" containerName="revoke-keys"
Nov 02 02:01:08.550594 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.550480    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc2408c-0e37-4e02-989a-3682209fadf6" containerName="revoke-keys"
Nov 02 02:01:08.550594 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.550539    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="8125b06a-f1eb-49de-80ac-61a79d4790c0" containerName="main"
Nov 02 02:01:08.550594 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.550547    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc2408c-0e37-4e02-989a-3682209fadf6" containerName="revoke-keys"
Nov 02 02:01:08.550594 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.550558    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c418ec3-c826-465a-bc5a-d73a9043ec8e" containerName="main"
Nov 02 02:01:08.553788 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.553769    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:08.564431 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.564399    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"]
Nov 02 02:01:08.572014 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.571494    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:01:08.572014 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.571579    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\""
Nov 02 02:01:08.572014 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.571581    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:01:08.575948 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.575929    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:01:08.648716 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.648669    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwzr\" (UniqueName: \"kubernetes.io/projected/5dae463f-239c-43e6-85be-0d6958a103ef-kube-api-access-hfwzr\") pod \"maas-api-revoke-sub-e87e1fd5-gbrnx\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:08.648903 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.648789    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e87e1fd5-gbrnx\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:08.648903 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.648857    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e87e1fd5-gbrnx\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:08.749604 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.749569    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e87e1fd5-gbrnx\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:08.749604 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.749616    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e87e1fd5-gbrnx\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:08.749883 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.749665    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwzr\" (UniqueName: \"kubernetes.io/projected/5dae463f-239c-43e6-85be-0d6958a103ef-kube-api-access-hfwzr\") pod \"maas-api-revoke-sub-e87e1fd5-gbrnx\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:08.750405 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.750379    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e87e1fd5-gbrnx\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:08.752250 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.752228    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e87e1fd5-gbrnx\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:08.774985 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.774948    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwzr\" (UniqueName: \"kubernetes.io/projected/5dae463f-239c-43e6-85be-0d6958a103ef-kube-api-access-hfwzr\") pod \"maas-api-revoke-sub-e87e1fd5-gbrnx\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:08.867567 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:08.867467    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:09.161912 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:09.161886    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"]
Nov 02 02:01:09.164983 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:01:09.164956    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dae463f_239c_43e6_85be_0d6958a103ef.slice/crio-598ae24ef0defd785659209c8fad4acf3ef85fb3bc1b08aee90e1b769fbe725a WatchSource:0}: Error finding container 598ae24ef0defd785659209c8fad4acf3ef85fb3bc1b08aee90e1b769fbe725a: Status 404 returned error can't find the container with id 598ae24ef0defd785659209c8fad4acf3ef85fb3bc1b08aee90e1b769fbe725a
Nov 02 02:01:09.648568 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:09.648534    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx" event={"ID":"5dae463f-239c-43e6-85be-0d6958a103ef","Type":"ContainerStarted","Data":"0d5675ffaaf217ce37e33dae5bc302ae70ea79236a8d20297016845a720e1a02"}
Nov 02 02:01:09.648996 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:09.648586    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx" event={"ID":"5dae463f-239c-43e6-85be-0d6958a103ef","Type":"ContainerStarted","Data":"598ae24ef0defd785659209c8fad4acf3ef85fb3bc1b08aee90e1b769fbe725a"}
Nov 02 02:01:09.689558 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:09.689434    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx" podStartSLOduration=1.689418649 podStartE2EDuration="1.689418649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:09.678163454 +0000 UTC m=+2774.983096280" watchObservedRunningTime="2026-11-02 02:01:09.689418649 +0000 UTC m=+2774.994351475"
Nov 02 02:01:10.654586 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:10.654551    4851 generic.go:356] "Generic (PLEG): container finished" podID="5dae463f-239c-43e6-85be-0d6958a103ef" containerID="0d5675ffaaf217ce37e33dae5bc302ae70ea79236a8d20297016845a720e1a02" exitCode=0
Nov 02 02:01:10.655014 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:10.654641    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx" event={"ID":"5dae463f-239c-43e6-85be-0d6958a103ef","Type":"ContainerDied","Data":"0d5675ffaaf217ce37e33dae5bc302ae70ea79236a8d20297016845a720e1a02"}
Nov 02 02:01:11.802182 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:11.802149    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:11.881993 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:11.881954    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dae463f-239c-43e6-85be-0d6958a103ef-kube-api-access-hfwzr\" (UniqueName: \"kubernetes.io/projected/5dae463f-239c-43e6-85be-0d6958a103ef-kube-api-access-hfwzr\") pod \"5dae463f-239c-43e6-85be-0d6958a103ef\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") "
Nov 02 02:01:11.882213 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:11.882014    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-xks-ca\") pod \"5dae463f-239c-43e6-85be-0d6958a103ef\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") "
Nov 02 02:01:11.882213 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:11.882067    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-openshift-ca\") pod \"5dae463f-239c-43e6-85be-0d6958a103ef\" (UID: \"5dae463f-239c-43e6-85be-0d6958a103ef\") "
Nov 02 02:01:11.882442 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:11.882406    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-openshift-ca" pod "5dae463f-239c-43e6-85be-0d6958a103ef" (UID: "5dae463f-239c-43e6-85be-0d6958a103ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:11.884380 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:11.884356    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-xks-ca" pod "5dae463f-239c-43e6-85be-0d6958a103ef" (UID: "5dae463f-239c-43e6-85be-0d6958a103ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:11.884519 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:11.884503    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dae463f-239c-43e6-85be-0d6958a103ef-kube-api-access-hfwzr" pod "5dae463f-239c-43e6-85be-0d6958a103ef" (UID: "5dae463f-239c-43e6-85be-0d6958a103ef"). InnerVolumeSpecName "kube-api-access-hfwzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:11.983855 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:11.983814    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-openshift-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:11.983855 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:11.983850    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfwzr\" (UniqueName: \"kubernetes.io/projected/5dae463f-239c-43e6-85be-0d6958a103ef-kube-api-access-hfwzr\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:11.983855 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:11.983864    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dae463f-239c-43e6-85be-0d6958a103ef-maas-api-xks-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:12.665759 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:12.665730    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"
Nov 02 02:01:12.665962 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:12.665778    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx" event={"ID":"5dae463f-239c-43e6-85be-0d6958a103ef","Type":"ContainerDied","Data":"598ae24ef0defd785659209c8fad4acf3ef85fb3bc1b08aee90e1b769fbe725a"}
Nov 02 02:01:12.665962 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:12.665829    4851 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="598ae24ef0defd785659209c8fad4acf3ef85fb3bc1b08aee90e1b769fbe725a"
Nov 02 02:01:17.656590 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:01:17.656550    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:01:20.894388 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.894355    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-97b27"]
Nov 02 02:01:20.894974 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.894951    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5dae463f-239c-43e6-85be-0d6958a103ef" containerName="revoke-keys"
Nov 02 02:01:20.895028 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.894979    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dae463f-239c-43e6-85be-0d6958a103ef" containerName="revoke-keys"
Nov 02 02:01:20.895199 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.895184    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dae463f-239c-43e6-85be-0d6958a103ef" containerName="revoke-keys"
Nov 02 02:01:20.897894 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.897875    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:20.908095 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.908053    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-97b27"]
Nov 02 02:01:20.911893 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.911867    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"]
Nov 02 02:01:20.914230 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.914207    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:20.916457 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.916432    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:01:20.916457 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.916432    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:01:20.919946 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.919915    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 02:01:20.920183 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.920162    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8qwrd\""
Nov 02 02:01:20.932057 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.928759    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 02:01:20.932318 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:20.932274    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"]
Nov 02 02:01:21.068115 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.068078    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2195022b-0d34-4c83-8a47-c8361b406708-metrics-certs\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.068115 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.068117    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbfc372-986d-4d79-995f-543f0172dbb4-openshift-service-ca\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.068402 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.068140    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2195022b-0d34-4c83-8a47-c8361b406708-openshift-service-ca\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.068402 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.068276    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2195022b-0d34-4c83-8a47-c8361b406708-maas-api-tls\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.068402 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.068340    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc86q\" (UniqueName: \"kubernetes.io/projected/2195022b-0d34-4c83-8a47-c8361b406708-kube-api-access-lc86q\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.068402 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.068398    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gvdd\" (UniqueName: \"kubernetes.io/projected/5cbfc372-986d-4d79-995f-543f0172dbb4-kube-api-access-9gvdd\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.068544 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.068463    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-metrics-certs\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.068544 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.068520    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-maas-api-tls\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.169448 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.169411    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2195022b-0d34-4c83-8a47-c8361b406708-openshift-service-ca\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.169645 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.169467    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2195022b-0d34-4c83-8a47-c8361b406708-maas-api-tls\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.169645 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.169484    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc86q\" (UniqueName: \"kubernetes.io/projected/2195022b-0d34-4c83-8a47-c8361b406708-kube-api-access-lc86q\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.169645 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.169523    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gvdd\" (UniqueName: \"kubernetes.io/projected/5cbfc372-986d-4d79-995f-543f0172dbb4-kube-api-access-9gvdd\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.169645 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.169561    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-metrics-certs\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.169645 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.169618    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-maas-api-tls\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.169926 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.169658    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2195022b-0d34-4c83-8a47-c8361b406708-metrics-certs\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.169926 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.169838    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbfc372-986d-4d79-995f-543f0172dbb4-openshift-service-ca\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.170543 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.170516    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2195022b-0d34-4c83-8a47-c8361b406708-openshift-service-ca\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.170850 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.170825    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbfc372-986d-4d79-995f-543f0172dbb4-openshift-service-ca\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.172407 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.172382    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-metrics-certs\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.172495 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.172483    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2195022b-0d34-4c83-8a47-c8361b406708-metrics-certs\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.172556 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.172536    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2195022b-0d34-4c83-8a47-c8361b406708-maas-api-tls\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.172601 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.172536    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-maas-api-tls\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.189656 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.189625    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc86q\" (UniqueName: \"kubernetes.io/projected/2195022b-0d34-4c83-8a47-c8361b406708-kube-api-access-lc86q\") pod \"maas-api-694bbcbf67-jwgdr\" (UID: \"2195022b-0d34-4c83-8a47-c8361b406708\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.189831 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.189805    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gvdd\" (UniqueName: \"kubernetes.io/projected/5cbfc372-986d-4d79-995f-543f0172dbb4-kube-api-access-9gvdd\") pod \"maas-api-6cff7899-97b27\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") " pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.211577 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.211537    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:21.229566 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.229542    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:21.507525 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.507497    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-97b27"]
Nov 02 02:01:21.510018 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:01:21.509977    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbfc372_986d_4d79_995f_543f0172dbb4.slice/crio-880c4e028a1137e060933a0ad8871f2a6f0f253261c09e3f296b0f1781a2fdae WatchSource:0}: Error finding container 880c4e028a1137e060933a0ad8871f2a6f0f253261c09e3f296b0f1781a2fdae: Status 404 returned error can't find the container with id 880c4e028a1137e060933a0ad8871f2a6f0f253261c09e3f296b0f1781a2fdae
Nov 02 02:01:21.511496 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.511476    4851 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:01:21.541403 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.541278    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"]
Nov 02 02:01:21.543692 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:01:21.543663    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2195022b_0d34_4c83_8a47_c8361b406708.slice/crio-16d5b92a41d8b09d89e49dea906dfc1bc7dcceb19f4e192e19ce2b40ab93ace1 WatchSource:0}: Error finding container 16d5b92a41d8b09d89e49dea906dfc1bc7dcceb19f4e192e19ce2b40ab93ace1: Status 404 returned error can't find the container with id 16d5b92a41d8b09d89e49dea906dfc1bc7dcceb19f4e192e19ce2b40ab93ace1
Nov 02 02:01:21.709261 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.709224    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr" event={"ID":"2195022b-0d34-4c83-8a47-c8361b406708","Type":"ContainerStarted","Data":"16d5b92a41d8b09d89e49dea906dfc1bc7dcceb19f4e192e19ce2b40ab93ace1"}
Nov 02 02:01:21.710531 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:21.710500    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27" event={"ID":"5cbfc372-986d-4d79-995f-543f0172dbb4","Type":"ContainerStarted","Data":"880c4e028a1137e060933a0ad8871f2a6f0f253261c09e3f296b0f1781a2fdae"}
Nov 02 02:01:22.717072 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:22.717021    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr" event={"ID":"2195022b-0d34-4c83-8a47-c8361b406708","Type":"ContainerStarted","Data":"3db7ba7bd689e28fa6c96a3303730bd6faac8e92184f9290b8304c4462f831e2"}
Nov 02 02:01:22.717583 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:22.717133    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:22.718707 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:22.718671    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27" event={"ID":"5cbfc372-986d-4d79-995f-543f0172dbb4","Type":"ContainerStarted","Data":"e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68"}
Nov 02 02:01:22.718851 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:22.718801    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:22.753815 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:22.753754    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr" podStartSLOduration=1.929256057 podStartE2EDuration="2.753739365s" totalImagesPullingTime="824.483308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:22.75352498 +0000 UTC m=+2788.058457806" watchObservedRunningTime="2026-11-02 02:01:22.753739365 +0000 UTC m=+2788.058672190"
Nov 02 02:01:22.790124 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:22.790071    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27" podStartSLOduration=2.161678262 podStartE2EDuration="2.790058417s" totalImagesPullingTime="628.380155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:22.788237504 +0000 UTC m=+2788.093170330" watchObservedRunningTime="2026-11-02 02:01:22.790058417 +0000 UTC m=+2788.094991593"
Nov 02 02:01:28.026357 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.026320    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"]
Nov 02 02:01:28.028788 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.028770    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.036055 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.036030    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ad5ce34c-openshift-default-dockercfg-vb2rj\""
Nov 02 02:01:28.037497 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.037471    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-istio-token\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.037637 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.037515    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b695db5-c451-4097-9f26-78a207bd499d-istiod-ca-cert\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.037701 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.037632    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-envoy\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.037701 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.037671    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-certs\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.037785 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.037753    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-socket\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.037825 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.037790    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b695db5-c451-4097-9f26-78a207bd499d-istio-podinfo\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.037825 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.037809    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qrc\" (UniqueName: \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-kube-api-access-j6qrc\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.037908 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.037829    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-credential-socket\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.037949 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.037911    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-data\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.043015 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.042986    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"]
Nov 02 02:01:28.139069 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139028    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-socket\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.139311 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139082    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b695db5-c451-4097-9f26-78a207bd499d-istio-podinfo\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.139311 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139111    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qrc\" (UniqueName: \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-kube-api-access-j6qrc\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.139311 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139142    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-credential-socket\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.139311 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139187    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-data\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.139311 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139249    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-istio-token\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.139625 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139324    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b695db5-c451-4097-9f26-78a207bd499d-istiod-ca-cert\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.139625 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139383    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-envoy\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.139625 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139407    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-certs\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.139625 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139518    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-socket\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.140096 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139918    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-certs\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.140096 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.139980    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-data\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.140573 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.140547    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-credential-socket\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.141256 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.141227    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b695db5-c451-4097-9f26-78a207bd499d-istiod-ca-cert\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.142470 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.142439    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b695db5-c451-4097-9f26-78a207bd499d-istio-podinfo\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.142938 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.142916    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-envoy\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.159801 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.159756    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-istio-token\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.160387 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.160355    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qrc\" (UniqueName: \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-kube-api-access-j6qrc\") pod \"e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") " pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.342776 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.342663    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:28.650952 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.650926    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"]
Nov 02 02:01:28.652829 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:01:28.652804    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b695db5_c451_4097_9f26_78a207bd499d.slice/crio-a1f131e31af795a92ec3f2cf37d10d91c3f88a3c2fa8a53783e8ead3827aae5a WatchSource:0}: Error finding container a1f131e31af795a92ec3f2cf37d10d91c3f88a3c2fa8a53783e8ead3827aae5a: Status 404 returned error can't find the container with id a1f131e31af795a92ec3f2cf37d10d91c3f88a3c2fa8a53783e8ead3827aae5a
Nov 02 02:01:28.654917 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.654887    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:01:28.655003 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.654951    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:01:28.655050 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.654999    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:01:28.729503 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.729473    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-jwgdr"
Nov 02 02:01:28.729943 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.729925    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:28.749004 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.748972    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" event={"ID":"7b695db5-c451-4097-9f26-78a207bd499d","Type":"ContainerStarted","Data":"9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95"}
Nov 02 02:01:28.749173 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.749023    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" event={"ID":"7b695db5-c451-4097-9f26-78a207bd499d","Type":"ContainerStarted","Data":"a1f131e31af795a92ec3f2cf37d10d91c3f88a3c2fa8a53783e8ead3827aae5a"}
Nov 02 02:01:28.799336 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.799267    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-97b27"]
Nov 02 02:01:28.799525 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.799502    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27" podUID="5cbfc372-986d-4d79-995f-543f0172dbb4" containerName="maas-api" containerID="cri-o://e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68" gracePeriod=30
Nov 02 02:01:28.855015 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:28.854948    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" podStartSLOduration=0.854924699 podStartE2EDuration="854.924699ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:28.850573339 +0000 UTC m=+2794.155506192" watchObservedRunningTime="2026-11-02 02:01:28.854924699 +0000 UTC m=+2794.159857526"
Nov 02 02:01:29.068388 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.068362    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:29.148534 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.148500    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbfc372-986d-4d79-995f-543f0172dbb4-kube-api-access-9gvdd\" (UniqueName: \"kubernetes.io/projected/5cbfc372-986d-4d79-995f-543f0172dbb4-kube-api-access-9gvdd\") pod \"5cbfc372-986d-4d79-995f-543f0172dbb4\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") "
Nov 02 02:01:29.148534 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.148535    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-metrics-certs\") pod \"5cbfc372-986d-4d79-995f-543f0172dbb4\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") "
Nov 02 02:01:29.148764 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.148592    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-maas-api-tls\") pod \"5cbfc372-986d-4d79-995f-543f0172dbb4\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") "
Nov 02 02:01:29.148764 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.148651    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbfc372-986d-4d79-995f-543f0172dbb4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbfc372-986d-4d79-995f-543f0172dbb4-openshift-service-ca\") pod \"5cbfc372-986d-4d79-995f-543f0172dbb4\" (UID: \"5cbfc372-986d-4d79-995f-543f0172dbb4\") "
Nov 02 02:01:29.149123 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.149082    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbfc372-986d-4d79-995f-543f0172dbb4-openshift-service-ca" pod "5cbfc372-986d-4d79-995f-543f0172dbb4" (UID: "5cbfc372-986d-4d79-995f-543f0172dbb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:29.150956 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.150932    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-maas-api-tls" pod "5cbfc372-986d-4d79-995f-543f0172dbb4" (UID: "5cbfc372-986d-4d79-995f-543f0172dbb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:29.151049 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.150953    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-metrics-certs" pod "5cbfc372-986d-4d79-995f-543f0172dbb4" (UID: "5cbfc372-986d-4d79-995f-543f0172dbb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:29.151049 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.151010    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbfc372-986d-4d79-995f-543f0172dbb4-kube-api-access-9gvdd" pod "5cbfc372-986d-4d79-995f-543f0172dbb4" (UID: "5cbfc372-986d-4d79-995f-543f0172dbb4"). InnerVolumeSpecName "kube-api-access-9gvdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:29.249739 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.249649    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-maas-api-tls\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:29.249739 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.249681    4851 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbfc372-986d-4d79-995f-543f0172dbb4-openshift-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:29.249739 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.249692    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gvdd\" (UniqueName: \"kubernetes.io/projected/5cbfc372-986d-4d79-995f-543f0172dbb4-kube-api-access-9gvdd\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:29.249739 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.249702    4851 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbfc372-986d-4d79-995f-543f0172dbb4-metrics-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:29.343884 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.343842    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:29.349054 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.349031    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:29.755005 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.754970    4851 generic.go:356] "Generic (PLEG): container finished" podID="5cbfc372-986d-4d79-995f-543f0172dbb4" containerID="e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68" exitCode=0
Nov 02 02:01:29.755200 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.755018    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27"
Nov 02 02:01:29.755200 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.755056    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27" event={"ID":"5cbfc372-986d-4d79-995f-543f0172dbb4","Type":"ContainerDied","Data":"e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68"}
Nov 02 02:01:29.755200 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.755109    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cff7899-97b27" event={"ID":"5cbfc372-986d-4d79-995f-543f0172dbb4","Type":"ContainerDied","Data":"880c4e028a1137e060933a0ad8871f2a6f0f253261c09e3f296b0f1781a2fdae"}
Nov 02 02:01:29.755200 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.755132    4851 scope.go:122] "RemoveContainer" containerID="e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68"
Nov 02 02:01:29.755514 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.755486    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:29.756733 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.756715    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:29.765057 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.765039    4851 scope.go:122] "RemoveContainer" containerID="e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68"
Nov 02 02:01:29.765374 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:01:29.765353    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68\": container with ID starting with e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68 not found: ID does not exist" containerID="e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68"
Nov 02 02:01:29.765439 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.765381    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68"} err="failed to get container status \"e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68\": rpc error: code = NotFound desc = could not find container \"e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68\": container with ID starting with e5addbdfe680a1704570bbfe4924ec152493bb679aa9ae84c9d29ddd1d330a68 not found: ID does not exist"
Nov 02 02:01:29.846990 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.846957    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-97b27"]
Nov 02 02:01:29.848953 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:29.848930    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cff7899-97b27"]
Nov 02 02:01:30.845467 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:30.845427    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbfc372-986d-4d79-995f-543f0172dbb4" path="/var/lib/kubelet/pods/5cbfc372-986d-4d79-995f-543f0172dbb4/volumes"
Nov 02 02:01:31.025434 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.025398    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"]
Nov 02 02:01:31.037082 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.037052    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b2dr"]
Nov 02 02:01:31.817742 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.817706    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"]
Nov 02 02:01:31.818128 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.818116    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5cbfc372-986d-4d79-995f-543f0172dbb4" containerName="maas-api"
Nov 02 02:01:31.818173 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.818130    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbfc372-986d-4d79-995f-543f0172dbb4" containerName="maas-api"
Nov 02 02:01:31.818227 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.818206    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbfc372-986d-4d79-995f-543f0172dbb4" containerName="maas-api"
Nov 02 02:01:31.829452 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.829413    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.834992 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.834930    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"]
Nov 02 02:01:31.840571 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.840525    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ad5ce34c-serving-cert\""
Nov 02 02:01:31.874958 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.874923    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a5b602-27a3-4aea-aef8-c1426a60161e-openshift-service-ca\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.875357 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.874986    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-maas-api-tls\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.875357 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.875092    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8mwd\" (UniqueName: \"kubernetes.io/projected/49a5b602-27a3-4aea-aef8-c1426a60161e-kube-api-access-g8mwd\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.875357 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.875133    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-metrics-certs\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.976622 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.976587    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a5b602-27a3-4aea-aef8-c1426a60161e-openshift-service-ca\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.976801 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.976639    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-maas-api-tls\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.976801 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.976665    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8mwd\" (UniqueName: \"kubernetes.io/projected/49a5b602-27a3-4aea-aef8-c1426a60161e-kube-api-access-g8mwd\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.976801 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.976689    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-metrics-certs\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.977416 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.977342    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a5b602-27a3-4aea-aef8-c1426a60161e-openshift-service-ca\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.979393 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.979372    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-metrics-certs\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.979483 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.979420    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-maas-api-tls\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:31.998922 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:31.998898    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8mwd\" (UniqueName: \"kubernetes.io/projected/49a5b602-27a3-4aea-aef8-c1426a60161e-kube-api-access-g8mwd\") pod \"maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:32.145841 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:32.145756    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:32.444546 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:32.444520    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"]
Nov 02 02:01:32.446542 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:01:32.446515    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a5b602_27a3_4aea_aef8_c1426a60161e.slice/crio-a401f754314269c78e4782ea9ff655e2db4813a676c3661bd13c3ecf7a846cc2 WatchSource:0}: Error finding container a401f754314269c78e4782ea9ff655e2db4813a676c3661bd13c3ecf7a846cc2: Status 404 returned error can't find the container with id a401f754314269c78e4782ea9ff655e2db4813a676c3661bd13c3ecf7a846cc2
Nov 02 02:01:32.772819 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:32.772734    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8" event={"ID":"49a5b602-27a3-4aea-aef8-c1426a60161e","Type":"ContainerStarted","Data":"a401f754314269c78e4782ea9ff655e2db4813a676c3661bd13c3ecf7a846cc2"}
Nov 02 02:01:32.845504 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:32.845439    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2517908f-0c91-42cd-bcf4-f17afb7059e5" path="/var/lib/kubelet/pods/2517908f-0c91-42cd-bcf4-f17afb7059e5/volumes"
Nov 02 02:01:33.352709 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.352623    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"]
Nov 02 02:01:33.354980 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.354963    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:01:33.369340 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.369274    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"]
Nov 02 02:01:33.378422 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.378395    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ad5ce34c\""
Nov 02 02:01:33.378422 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.378427    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ad5ce34c-dockercfg-wrhfx\""
Nov 02 02:01:33.388893 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.388860    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f6xf\" (UniqueName: \"kubernetes.io/projected/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-kube-api-access-8f6xf\") pod \"payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd\" (UID: \"fb41ed37-cf39-49e5-85e4-5dc0bc70f324\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:01:33.389051 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.388912    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd\" (UID: \"fb41ed37-cf39-49e5-85e4-5dc0bc70f324\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:01:33.464765 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.464724    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"]
Nov 02 02:01:33.467155 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.467139    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:01:33.480518 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.480490    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"]
Nov 02 02:01:33.489659 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.489628    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f6xf\" (UniqueName: \"kubernetes.io/projected/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-kube-api-access-8f6xf\") pod \"payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd\" (UID: \"fb41ed37-cf39-49e5-85e4-5dc0bc70f324\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:01:33.489845 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.489700    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht\" (UID: \"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f\") " pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:01:33.489845 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.489738    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd\" (UID: \"fb41ed37-cf39-49e5-85e4-5dc0bc70f324\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:01:33.489845 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.489778    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscn4\" (UniqueName: \"kubernetes.io/projected/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-kube-api-access-bscn4\") pod \"payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht\" (UID: \"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f\") " pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:01:33.490545 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.490523    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd\" (UID: \"fb41ed37-cf39-49e5-85e4-5dc0bc70f324\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:01:33.510636 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.510611    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f6xf\" (UniqueName: \"kubernetes.io/projected/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-kube-api-access-8f6xf\") pod \"payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd\" (UID: \"fb41ed37-cf39-49e5-85e4-5dc0bc70f324\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:01:33.591168 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.591132    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht\" (UID: \"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f\") " pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:01:33.591390 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.591182    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscn4\" (UniqueName: \"kubernetes.io/projected/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-kube-api-access-bscn4\") pod \"payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht\" (UID: \"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f\") " pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:01:33.591855 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.591834    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht\" (UID: \"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f\") " pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:01:33.613648 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.613576    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscn4\" (UniqueName: \"kubernetes.io/projected/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-kube-api-access-bscn4\") pod \"payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht\" (UID: \"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f\") " pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:01:33.667123 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.667081    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:01:33.778697 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.778669    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:01:33.779599 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.779572    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8" event={"ID":"49a5b602-27a3-4aea-aef8-c1426a60161e","Type":"ContainerStarted","Data":"8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315"}
Nov 02 02:01:33.779709 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.779690    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:33.814364 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.814219    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8" podStartSLOduration=2.267090605 podStartE2EDuration="2.81420101s" totalImagesPullingTime="547.110405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:33.813741186 +0000 UTC m=+2799.118674040" watchObservedRunningTime="2026-11-02 02:01:33.81420101 +0000 UTC m=+2799.119133839"
Nov 02 02:01:33.963323 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:33.963279    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"]
Nov 02 02:01:33.965655 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:01:33.965626    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb41ed37_cf39_49e5_85e4_5dc0bc70f324.slice/crio-10f6e5182746c8c523d9aeecc187da3bbf4b12ee34d1de549169146585073f84 WatchSource:0}: Error finding container 10f6e5182746c8c523d9aeecc187da3bbf4b12ee34d1de549169146585073f84: Status 404 returned error can't find the container with id 10f6e5182746c8c523d9aeecc187da3bbf4b12ee34d1de549169146585073f84
Nov 02 02:01:34.120730 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:34.120701    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"]
Nov 02 02:01:34.122798 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:01:34.122771    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea45f57_07ff_4a41_b24e_02b6ccdfcd1f.slice/crio-bf7d0251870d2de9f7693cae7490b09d18d7a6bebe955f332896b9b7b4c98ed1 WatchSource:0}: Error finding container bf7d0251870d2de9f7693cae7490b09d18d7a6bebe955f332896b9b7b4c98ed1: Status 404 returned error can't find the container with id bf7d0251870d2de9f7693cae7490b09d18d7a6bebe955f332896b9b7b4c98ed1
Nov 02 02:01:34.786594 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:34.786552    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht" event={"ID":"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f","Type":"ContainerStarted","Data":"42909d8abb3cd975e2d6a8d1e9241e37594d04356b0d40711726baca73bd5c8a"}
Nov 02 02:01:34.787081 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:34.786617    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht" event={"ID":"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f","Type":"ContainerStarted","Data":"bf7d0251870d2de9f7693cae7490b09d18d7a6bebe955f332896b9b7b4c98ed1"}
Nov 02 02:01:34.787081 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:34.786777    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:01:34.788472 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:34.788444    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd" event={"ID":"fb41ed37-cf39-49e5-85e4-5dc0bc70f324","Type":"ContainerStarted","Data":"b2a95a2d2a7bd394bbb2f6fa757cfc4d3407dccef83c9f895c8840e6591f0475"}
Nov 02 02:01:34.788612 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:34.788494    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd" event={"ID":"fb41ed37-cf39-49e5-85e4-5dc0bc70f324","Type":"ContainerStarted","Data":"10f6e5182746c8c523d9aeecc187da3bbf4b12ee34d1de549169146585073f84"}
Nov 02 02:01:34.788765 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:34.788741    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:01:34.827938 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:34.827873    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht" podStartSLOduration=1.827856379 podStartE2EDuration="1.827856379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:34.825568378 +0000 UTC m=+2800.130501206" watchObservedRunningTime="2026-11-02 02:01:34.827856379 +0000 UTC m=+2800.132789205"
Nov 02 02:01:34.865669 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:34.865616    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd" podStartSLOduration=1.865602817 podStartE2EDuration="1.865602817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:34.864032366 +0000 UTC m=+2800.168965194" watchObservedRunningTime="2026-11-02 02:01:34.865602817 +0000 UTC m=+2800.170535643"
Nov 02 02:01:39.794086 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:39.794056    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:44.757131 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:44.757097    4851 patch_prober.go:29] interesting pod/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 02:01:44.757607 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:44.757152    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 02:01:44.841517 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:44.841483    4851 generic.go:356] "Generic (PLEG): container finished" podID="7b695db5-c451-4097-9f26-78a207bd499d" containerID="9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95" exitCode=0
Nov 02 02:01:44.844850 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:44.844826    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" event={"ID":"7b695db5-c451-4097-9f26-78a207bd499d","Type":"ContainerDied","Data":"9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95"}
Nov 02 02:01:44.845209 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:44.845193    4851 scope.go:122] "RemoveContainer" containerID="9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95"
Nov 02 02:01:44.846269 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:44.846245    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:01:44.846343 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:44.846331    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:01:44.846383 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:44.846365    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:01:45.796043 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:45.796016    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:01:45.796515 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:45.796170    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:01:45.870396 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:45.870358    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" event={"ID":"7b695db5-c451-4097-9f26-78a207bd499d","Type":"ContainerStarted","Data":"580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6"}
Nov 02 02:01:46.352316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:46.348754    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:46.352316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:46.348885    4851 patch_prober.go:29] interesting pod/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 02:01:46.352316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:46.348928    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 02:01:47.345748 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:47.345711    4851 patch_prober.go:29] interesting pod/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 02:01:47.346398 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:47.346363    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 02:01:47.678422 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:01:47.678387    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:01:48.356317 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:48.354755    4851 patch_prober.go:29] interesting pod/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 02:01:48.356317 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:48.354815    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 02:01:49.347783 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:49.347744    4851 patch_prober.go:29] interesting pod/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 02:01:49.348097 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:49.348066    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 02:01:50.351315 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:50.350713    4851 patch_prober.go:29] interesting pod/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 02:01:50.351315 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:50.350785    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 02:01:51.347753 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:51.347724    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:51.348249 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:51.348233    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:51.348864 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:51.348845    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:01:56.762732 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:56.762686    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"]
Nov 02 02:01:56.809395 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:56.809358    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"]
Nov 02 02:01:57.533695 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.533649    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"]
Nov 02 02:01:57.534813 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.534769    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8" podUID="49a5b602-27a3-4aea-aef8-c1426a60161e" containerName="maas-api" containerID="cri-o://8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315" gracePeriod=30
Nov 02 02:01:57.814541 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.814514    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:57.872852 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.872813    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a5b602-27a3-4aea-aef8-c1426a60161e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a5b602-27a3-4aea-aef8-c1426a60161e-openshift-service-ca\") pod \"49a5b602-27a3-4aea-aef8-c1426a60161e\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") "
Nov 02 02:01:57.873065 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.872867    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-metrics-certs\") pod \"49a5b602-27a3-4aea-aef8-c1426a60161e\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") "
Nov 02 02:01:57.873065 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.872929    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-maas-api-tls\") pod \"49a5b602-27a3-4aea-aef8-c1426a60161e\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") "
Nov 02 02:01:57.873065 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.872981    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a5b602-27a3-4aea-aef8-c1426a60161e-kube-api-access-g8mwd\" (UniqueName: \"kubernetes.io/projected/49a5b602-27a3-4aea-aef8-c1426a60161e-kube-api-access-g8mwd\") pod \"49a5b602-27a3-4aea-aef8-c1426a60161e\" (UID: \"49a5b602-27a3-4aea-aef8-c1426a60161e\") "
Nov 02 02:01:57.873439 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.873405    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a5b602-27a3-4aea-aef8-c1426a60161e-openshift-service-ca" pod "49a5b602-27a3-4aea-aef8-c1426a60161e" (UID: "49a5b602-27a3-4aea-aef8-c1426a60161e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:57.875375 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.875343    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-metrics-certs" pod "49a5b602-27a3-4aea-aef8-c1426a60161e" (UID: "49a5b602-27a3-4aea-aef8-c1426a60161e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:57.875874 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.875848    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-maas-api-tls" pod "49a5b602-27a3-4aea-aef8-c1426a60161e" (UID: "49a5b602-27a3-4aea-aef8-c1426a60161e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:57.875975 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.875869    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a5b602-27a3-4aea-aef8-c1426a60161e-kube-api-access-g8mwd" pod "49a5b602-27a3-4aea-aef8-c1426a60161e" (UID: "49a5b602-27a3-4aea-aef8-c1426a60161e"). InnerVolumeSpecName "kube-api-access-g8mwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:57.937338 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.937276    4851 generic.go:356] "Generic (PLEG): container finished" podID="49a5b602-27a3-4aea-aef8-c1426a60161e" containerID="8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315" exitCode=0
Nov 02 02:01:57.937527 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.937344    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"
Nov 02 02:01:57.937527 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.937362    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8" event={"ID":"49a5b602-27a3-4aea-aef8-c1426a60161e","Type":"ContainerDied","Data":"8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315"}
Nov 02 02:01:57.937527 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.937422    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8" event={"ID":"49a5b602-27a3-4aea-aef8-c1426a60161e","Type":"ContainerDied","Data":"a401f754314269c78e4782ea9ff655e2db4813a676c3661bd13c3ecf7a846cc2"}
Nov 02 02:01:57.937527 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.937453    4851 scope.go:122] "RemoveContainer" containerID="8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315"
Nov 02 02:01:57.954233 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.954211    4851 scope.go:122] "RemoveContainer" containerID="8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315"
Nov 02 02:01:57.954630 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:01:57.954599    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315\": container with ID starting with 8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315 not found: ID does not exist" containerID="8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315"
Nov 02 02:01:57.954709 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.954641    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315"} err="failed to get container status \"8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315\": rpc error: code = NotFound desc = could not find container \"8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315\": container with ID starting with 8c37de721f2cf04c188c94fc3232fb903024242f46f5e3464e24bd8ddf332315 not found: ID does not exist"
Nov 02 02:01:57.968726 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.968690    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"]
Nov 02 02:01:57.971944 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.971918    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad5ce34c-7cf85696b5-4b2w8"]
Nov 02 02:01:57.974062 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.974039    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8mwd\" (UniqueName: \"kubernetes.io/projected/49a5b602-27a3-4aea-aef8-c1426a60161e-kube-api-access-g8mwd\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:57.974165 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.974066    4851 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49a5b602-27a3-4aea-aef8-c1426a60161e-openshift-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:57.974165 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.974079    4851 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-metrics-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:57.974165 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:57.974089    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49a5b602-27a3-4aea-aef8-c1426a60161e-maas-api-tls\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:01:58.844944 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:01:58.844915    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a5b602-27a3-4aea-aef8-c1426a60161e" path="/var/lib/kubelet/pods/49a5b602-27a3-4aea-aef8-c1426a60161e/volumes"
Nov 02 02:02:01.763938 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:01.763896    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht" podUID="cea45f57-07ff-4a41-b24e-02b6ccdfcd1f" containerName="payload-processing" containerID="cri-o://42909d8abb3cd975e2d6a8d1e9241e37594d04356b0d40711726baca73bd5c8a" gracePeriod=25
Nov 02 02:02:01.810644 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:01.810603    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd" podUID="fb41ed37-cf39-49e5-85e4-5dc0bc70f324" containerName="payload-pre-processing" containerID="cri-o://b2a95a2d2a7bd394bbb2f6fa757cfc4d3407dccef83c9f895c8840e6591f0475" gracePeriod=25
Nov 02 02:02:01.966704 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:01.966667    4851 generic.go:356] "Generic (PLEG): container finished" podID="cea45f57-07ff-4a41-b24e-02b6ccdfcd1f" containerID="42909d8abb3cd975e2d6a8d1e9241e37594d04356b0d40711726baca73bd5c8a" exitCode=0
Nov 02 02:02:01.966906 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:01.966755    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht" event={"ID":"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f","Type":"ContainerDied","Data":"42909d8abb3cd975e2d6a8d1e9241e37594d04356b0d40711726baca73bd5c8a"}
Nov 02 02:02:01.968255 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:01.968227    4851 generic.go:356] "Generic (PLEG): container finished" podID="fb41ed37-cf39-49e5-85e4-5dc0bc70f324" containerID="b2a95a2d2a7bd394bbb2f6fa757cfc4d3407dccef83c9f895c8840e6591f0475" exitCode=0
Nov 02 02:02:01.968441 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:01.968375    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd" event={"ID":"fb41ed37-cf39-49e5-85e4-5dc0bc70f324","Type":"ContainerDied","Data":"b2a95a2d2a7bd394bbb2f6fa757cfc4d3407dccef83c9f895c8840e6591f0475"}
Nov 02 02:02:02.063693 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.063669    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:02:02.094762 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.094738    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:02:02.114748 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.114712    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-plugins-config-volume\") pod \"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f\" (UID: \"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f\") "
Nov 02 02:02:02.114926 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.114798    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-kube-api-access-8f6xf\" (UniqueName: \"kubernetes.io/projected/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-kube-api-access-8f6xf\") pod \"fb41ed37-cf39-49e5-85e4-5dc0bc70f324\" (UID: \"fb41ed37-cf39-49e5-85e4-5dc0bc70f324\") "
Nov 02 02:02:02.114926 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.114830    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-plugins-config-volume\") pod \"fb41ed37-cf39-49e5-85e4-5dc0bc70f324\" (UID: \"fb41ed37-cf39-49e5-85e4-5dc0bc70f324\") "
Nov 02 02:02:02.114926 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.114864    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-kube-api-access-bscn4\" (UniqueName: \"kubernetes.io/projected/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-kube-api-access-bscn4\") pod \"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f\" (UID: \"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f\") "
Nov 02 02:02:02.115175 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.115148    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-plugins-config-volume" pod "cea45f57-07ff-4a41-b24e-02b6ccdfcd1f" (UID: "cea45f57-07ff-4a41-b24e-02b6ccdfcd1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:02.115256 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.115222    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-plugins-config-volume" pod "fb41ed37-cf39-49e5-85e4-5dc0bc70f324" (UID: "fb41ed37-cf39-49e5-85e4-5dc0bc70f324"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:02.117182 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.117158    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-kube-api-access-bscn4" pod "cea45f57-07ff-4a41-b24e-02b6ccdfcd1f" (UID: "cea45f57-07ff-4a41-b24e-02b6ccdfcd1f"). InnerVolumeSpecName "kube-api-access-bscn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:02.117182 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.117171    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-kube-api-access-8f6xf" pod "fb41ed37-cf39-49e5-85e4-5dc0bc70f324" (UID: "fb41ed37-cf39-49e5-85e4-5dc0bc70f324"). InnerVolumeSpecName "kube-api-access-8f6xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:02.216737 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.216706    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.216737 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.216733    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f6xf\" (UniqueName: \"kubernetes.io/projected/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-kube-api-access-8f6xf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.216737 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.216744    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb41ed37-cf39-49e5-85e4-5dc0bc70f324-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.216979 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.216753    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscn4\" (UniqueName: \"kubernetes.io/projected/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f-kube-api-access-bscn4\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.511691 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.511652    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"]
Nov 02 02:02:02.511948 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.511921    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy" containerID="cri-o://580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6" gracePeriod=30
Nov 02 02:02:02.975320 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.975255    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht" event={"ID":"cea45f57-07ff-4a41-b24e-02b6ccdfcd1f","Type":"ContainerDied","Data":"bf7d0251870d2de9f7693cae7490b09d18d7a6bebe955f332896b9b7b4c98ed1"}
Nov 02 02:02:02.975830 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.975351    4851 scope.go:122] "RemoveContainer" containerID="42909d8abb3cd975e2d6a8d1e9241e37594d04356b0d40711726baca73bd5c8a"
Nov 02 02:02:02.975830 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.975272    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"
Nov 02 02:02:02.976773 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.976737    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"
Nov 02 02:02:02.976773 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.976756    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd" event={"ID":"fb41ed37-cf39-49e5-85e4-5dc0bc70f324","Type":"ContainerDied","Data":"10f6e5182746c8c523d9aeecc187da3bbf4b12ee34d1de549169146585073f84"}
Nov 02 02:02:02.985256 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:02.985235    4851 scope.go:122] "RemoveContainer" containerID="b2a95a2d2a7bd394bbb2f6fa757cfc4d3407dccef83c9f895c8840e6591f0475"
Nov 02 02:02:03.014016 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:03.013994    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"]
Nov 02 02:02:03.018549 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:03.018527    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad5ce34c-6f49dddcf-5q9ht"]
Nov 02 02:02:03.049222 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:03.049165    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"]
Nov 02 02:02:03.053907 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:03.053888    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad5ce34c-57778f74f9-b9wdd"]
Nov 02 02:02:04.236271 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.236237    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"]
Nov 02 02:02:04.236689 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.236654    4851 cpu_manager.go:414] "containerMap: removing container" podUID="fb41ed37-cf39-49e5-85e4-5dc0bc70f324" containerName="payload-pre-processing"
Nov 02 02:02:04.236689 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.236667    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb41ed37-cf39-49e5-85e4-5dc0bc70f324" containerName="payload-pre-processing"
Nov 02 02:02:04.236689 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.236677    4851 cpu_manager.go:414] "containerMap: removing container" podUID="cea45f57-07ff-4a41-b24e-02b6ccdfcd1f" containerName="payload-processing"
Nov 02 02:02:04.236689 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.236683    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea45f57-07ff-4a41-b24e-02b6ccdfcd1f" containerName="payload-processing"
Nov 02 02:02:04.236837 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.236698    4851 cpu_manager.go:414] "containerMap: removing container" podUID="49a5b602-27a3-4aea-aef8-c1426a60161e" containerName="maas-api"
Nov 02 02:02:04.236837 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.236703    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a5b602-27a3-4aea-aef8-c1426a60161e" containerName="maas-api"
Nov 02 02:02:04.236837 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.236769    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb41ed37-cf39-49e5-85e4-5dc0bc70f324" containerName="payload-pre-processing"
Nov 02 02:02:04.236837 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.236777    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea45f57-07ff-4a41-b24e-02b6ccdfcd1f" containerName="payload-processing"
Nov 02 02:02:04.236837 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.236785    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a5b602-27a3-4aea-aef8-c1426a60161e" containerName="maas-api"
Nov 02 02:02:04.241431 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.241413    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.258112 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.258084    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"]
Nov 02 02:02:04.268645 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.268618    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-b12eb5-openshift-default-dockercfg-6jlz9\""
Nov 02 02:02:04.338939 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.338907    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-socket\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.339089 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.338948    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wnj5\" (UniqueName: \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-kube-api-access-9wnj5\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.339089 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.338970    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558da18b-bd75-4787-8619-1a4851b037bb-istiod-ca-cert\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.339089 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.339062    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-credential-socket\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.339089 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.339083    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-istio-token\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.339242 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.339151    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-envoy\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.339242 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.339203    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-certs\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.339242 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.339223    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-data\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.339364 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.339259    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558da18b-bd75-4787-8619-1a4851b037bb-istio-podinfo\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.439989 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.439950    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-socket\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.439989 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.439992    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wnj5\" (UniqueName: \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-kube-api-access-9wnj5\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440242 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440012    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558da18b-bd75-4787-8619-1a4851b037bb-istiod-ca-cert\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440242 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440039    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-credential-socket\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440242 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440156    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-istio-token\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440242 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440187    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-envoy\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440542 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440248    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-certs\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440542 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440272    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-data\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440542 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440363    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558da18b-bd75-4787-8619-1a4851b037bb-istio-podinfo\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440542 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440428    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-socket\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440767 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440612    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-certs\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440767 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440657    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-credential-socket\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440767 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440693    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-data\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.440939 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.440831    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558da18b-bd75-4787-8619-1a4851b037bb-istiod-ca-cert\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.442685 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.442661    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-envoy\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.443002 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.442979    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558da18b-bd75-4787-8619-1a4851b037bb-istio-podinfo\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.464999 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.464974    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-istio-token\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.465182 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.465161    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wnj5\" (UniqueName: \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-kube-api-access-9wnj5\") pod \"e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") " pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.555148 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.555072    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:04.845301 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.845207    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea45f57-07ff-4a41-b24e-02b6ccdfcd1f" path="/var/lib/kubelet/pods/cea45f57-07ff-4a41-b24e-02b6ccdfcd1f/volumes"
Nov 02 02:02:04.845640 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.845624    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb41ed37-cf39-49e5-85e4-5dc0bc70f324" path="/var/lib/kubelet/pods/fb41ed37-cf39-49e5-85e4-5dc0bc70f324/volumes"
Nov 02 02:02:04.892992 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.892955    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"]
Nov 02 02:02:04.896713 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:02:04.896682    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558da18b_bd75_4787_8619_1a4851b037bb.slice/crio-8295d6dca158c8425b902d1d1242234e032a2767ff7b49f493d953ac9aee3415 WatchSource:0}: Error finding container 8295d6dca158c8425b902d1d1242234e032a2767ff7b49f493d953ac9aee3415: Status 404 returned error can't find the container with id 8295d6dca158c8425b902d1d1242234e032a2767ff7b49f493d953ac9aee3415
Nov 02 02:02:04.899504 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.899470    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:02:04.899627 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.899552    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:02:04.899627 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.899599    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:02:04.992181 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.992101    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" event={"ID":"558da18b-bd75-4787-8619-1a4851b037bb","Type":"ContainerStarted","Data":"62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e"}
Nov 02 02:02:04.992181 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:04.992154    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" event={"ID":"558da18b-bd75-4787-8619-1a4851b037bb","Type":"ContainerStarted","Data":"8295d6dca158c8425b902d1d1242234e032a2767ff7b49f493d953ac9aee3415"}
Nov 02 02:02:05.051375 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:05.051317    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" podStartSLOduration=1.051265631 podStartE2EDuration="1.051265631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:05.049482625 +0000 UTC m=+2830.354415464" watchObservedRunningTime="2026-11-02 02:02:05.051265631 +0000 UTC m=+2830.356198457"
Nov 02 02:02:05.555815 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:05.555777    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:05.561397 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:05.561368    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:05.997367 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:05.997332    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:05.998428 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:05.998404    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:06.350410 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:06.350317    4851 patch_prober.go:29] interesting pod/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 02:02:06.350410 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:06.350377    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:02:07.777463 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.777436    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:02:07.872588 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872497    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-envoy\") pod \"7b695db5-c451-4097-9f26-78a207bd499d\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") "
Nov 02 02:02:07.872749 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872591    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-certs\") pod \"7b695db5-c451-4097-9f26-78a207bd499d\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") "
Nov 02 02:02:07.872749 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872618    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-data\") pod \"7b695db5-c451-4097-9f26-78a207bd499d\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") "
Nov 02 02:02:07.872749 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872646    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b695db5-c451-4097-9f26-78a207bd499d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b695db5-c451-4097-9f26-78a207bd499d-istiod-ca-cert\") pod \"7b695db5-c451-4097-9f26-78a207bd499d\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") "
Nov 02 02:02:07.872749 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872664    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b695db5-c451-4097-9f26-78a207bd499d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b695db5-c451-4097-9f26-78a207bd499d-istio-podinfo\") pod \"7b695db5-c451-4097-9f26-78a207bd499d\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") "
Nov 02 02:02:07.872749 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872685    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-istio-token\" (UniqueName: \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-istio-token\") pod \"7b695db5-c451-4097-9f26-78a207bd499d\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") "
Nov 02 02:02:07.873039 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872859    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-kube-api-access-j6qrc\" (UniqueName: \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-kube-api-access-j6qrc\") pod \"7b695db5-c451-4097-9f26-78a207bd499d\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") "
Nov 02 02:02:07.873039 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872911    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-certs" pod "7b695db5-c451-4097-9f26-78a207bd499d" (UID: "7b695db5-c451-4097-9f26-78a207bd499d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:07.873039 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872930    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-data" pod "7b695db5-c451-4097-9f26-78a207bd499d" (UID: "7b695db5-c451-4097-9f26-78a207bd499d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:07.873039 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872955    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-socket\") pod \"7b695db5-c451-4097-9f26-78a207bd499d\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") "
Nov 02 02:02:07.873039 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.872994    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-credential-socket\") pod \"7b695db5-c451-4097-9f26-78a207bd499d\" (UID: \"7b695db5-c451-4097-9f26-78a207bd499d\") "
Nov 02 02:02:07.873387 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.873102    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b695db5-c451-4097-9f26-78a207bd499d-istiod-ca-cert" pod "7b695db5-c451-4097-9f26-78a207bd499d" (UID: "7b695db5-c451-4097-9f26-78a207bd499d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:07.873387 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.873222    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-socket" pod "7b695db5-c451-4097-9f26-78a207bd499d" (UID: "7b695db5-c451-4097-9f26-78a207bd499d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:07.873665 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.873495    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-credential-socket" pod "7b695db5-c451-4097-9f26-78a207bd499d" (UID: "7b695db5-c451-4097-9f26-78a207bd499d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:07.873759 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.873735    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:07.873830 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.873764    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:07.873830 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.873781    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b695db5-c451-4097-9f26-78a207bd499d-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:07.873830 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.873794    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:07.873830 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.873810    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:07.875143 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.875116    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-envoy" pod "7b695db5-c451-4097-9f26-78a207bd499d" (UID: "7b695db5-c451-4097-9f26-78a207bd499d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:07.875143 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.875130    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-istio-token" pod "7b695db5-c451-4097-9f26-78a207bd499d" (UID: "7b695db5-c451-4097-9f26-78a207bd499d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:07.875356 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.875192    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b695db5-c451-4097-9f26-78a207bd499d-istio-podinfo" pod "7b695db5-c451-4097-9f26-78a207bd499d" (UID: "7b695db5-c451-4097-9f26-78a207bd499d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:07.875356 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.875352    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-kube-api-access-j6qrc" pod "7b695db5-c451-4097-9f26-78a207bd499d" (UID: "7b695db5-c451-4097-9f26-78a207bd499d"). InnerVolumeSpecName "kube-api-access-j6qrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:07.974972 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.974941    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6qrc\" (UniqueName: \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-kube-api-access-j6qrc\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:07.974972 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.974971    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b695db5-c451-4097-9f26-78a207bd499d-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:07.974972 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.974981    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b695db5-c451-4097-9f26-78a207bd499d-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:07.975221 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:07.974990    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b695db5-c451-4097-9f26-78a207bd499d-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:02:08.009038 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.009007    4851 generic.go:356] "Generic (PLEG): container finished" podID="7b695db5-c451-4097-9f26-78a207bd499d" containerID="580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6" exitCode=0
Nov 02 02:02:08.009200 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.009058    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"
Nov 02 02:02:08.009200 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.009099    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" event={"ID":"7b695db5-c451-4097-9f26-78a207bd499d","Type":"ContainerDied","Data":"580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6"}
Nov 02 02:02:08.009200 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.009161    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b" event={"ID":"7b695db5-c451-4097-9f26-78a207bd499d","Type":"ContainerDied","Data":"a1f131e31af795a92ec3f2cf37d10d91c3f88a3c2fa8a53783e8ead3827aae5a"}
Nov 02 02:02:08.009200 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.009188    4851 scope.go:122] "RemoveContainer" containerID="580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6"
Nov 02 02:02:08.019015 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.018997    4851 scope.go:122] "RemoveContainer" containerID="9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95"
Nov 02 02:02:08.028107 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.028088    4851 scope.go:122] "RemoveContainer" containerID="580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6"
Nov 02 02:02:08.028401 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:02:08.028380    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6\": container with ID starting with 580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6 not found: ID does not exist" containerID="580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6"
Nov 02 02:02:08.028457 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.028410    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6"} err="failed to get container status \"580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6\": rpc error: code = NotFound desc = could not find container \"580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6\": container with ID starting with 580fae1f8d5dd6409dd75ebc448a44557780b41941325811c38e16073247f4a6 not found: ID does not exist"
Nov 02 02:02:08.028457 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.028432    4851 scope.go:122] "RemoveContainer" containerID="9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95"
Nov 02 02:02:08.028701 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:02:08.028682    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95\": container with ID starting with 9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95 not found: ID does not exist" containerID="9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95"
Nov 02 02:02:08.028751 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.028707    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95"} err="failed to get container status \"9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95\": rpc error: code = NotFound desc = could not find container \"9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95\": container with ID starting with 9dae6f27cfe66827b8891530e0c73b17cd06456f2da9e4b0b2c71035c6f0ca95 not found: ID does not exist"
Nov 02 02:02:08.039429 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.039407    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"]
Nov 02 02:02:08.043506 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.043486    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ad5ce34c-openshift-default-57746f445c-pnj2b"]
Nov 02 02:02:08.473358 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.473317    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"]
Nov 02 02:02:08.474087 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.474061    4851 cpu_manager.go:414] "containerMap: removing container" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy"
Nov 02 02:02:08.474135 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.474093    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy"
Nov 02 02:02:08.474135 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.474114    4851 cpu_manager.go:414] "containerMap: removing container" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy"
Nov 02 02:02:08.474135 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.474122    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy"
Nov 02 02:02:08.474269 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.474253    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy"
Nov 02 02:02:08.474324 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.474275    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b695db5-c451-4097-9f26-78a207bd499d" containerName="istio-proxy"
Nov 02 02:02:08.479738 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.479709    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.515209 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.515179    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-b12eb5-serving-cert\""
Nov 02 02:02:08.534233 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.534193    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"]
Nov 02 02:02:08.581298 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.581264    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.581493 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.581326    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.581493 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.581419    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.581493 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.581459    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mph5h\" (UniqueName: \"kubernetes.io/projected/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-kube-api-access-mph5h\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.682951 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.682915    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.682951 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.682957    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.683226 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.682982    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.683226 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.683019    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mph5h\" (UniqueName: \"kubernetes.io/projected/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-kube-api-access-mph5h\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.683785 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.683754    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.685930 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.685906    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.686042 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.685902    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.710862 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.710828    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mph5h\" (UniqueName: \"kubernetes.io/projected/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-kube-api-access-mph5h\") pod \"maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.794113 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.794025    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:08.848806 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:08.848594    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b695db5-c451-4097-9f26-78a207bd499d" path="/var/lib/kubelet/pods/7b695db5-c451-4097-9f26-78a207bd499d/volumes"
Nov 02 02:02:09.112382 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.110379    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"]
Nov 02 02:02:09.112740 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:02:09.112715    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce71f335_11a3_4c93_b7ed_efd7e001dfdc.slice/crio-5049265f2e5ecc51c3b0d92808e8b886b80d23d3010040b1edfb864b4a4fa7e2 WatchSource:0}: Error finding container 5049265f2e5ecc51c3b0d92808e8b886b80d23d3010040b1edfb864b4a4fa7e2: Status 404 returned error can't find the container with id 5049265f2e5ecc51c3b0d92808e8b886b80d23d3010040b1edfb864b4a4fa7e2
Nov 02 02:02:09.115800 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.115779    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"]
Nov 02 02:02:09.115900 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.115879    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:02:09.123579 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.123554    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"]
Nov 02 02:02:09.129096 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.129073    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-b12eb5\""
Nov 02 02:02:09.129262 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.129241    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-b12eb5-dockercfg-lg6df\""
Nov 02 02:02:09.187899 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.187870    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m9jc\" (UniqueName: \"kubernetes.io/projected/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-kube-api-access-9m9jc\") pod \"payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf\" (UID: \"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:02:09.188084 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.188002    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf\" (UID: \"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:02:09.197024 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.196988    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"]
Nov 02 02:02:09.200650 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.200631    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:02:09.212132 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.212105    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"]
Nov 02 02:02:09.288657 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.288622    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s5wf\" (UniqueName: \"kubernetes.io/projected/b886216b-7517-49cc-8406-1f13632adbe9-kube-api-access-7s5wf\") pod \"payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct\" (UID: \"b886216b-7517-49cc-8406-1f13632adbe9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:02:09.288801 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.288668    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf\" (UID: \"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:02:09.288856 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.288788    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b886216b-7517-49cc-8406-1f13632adbe9-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct\" (UID: \"b886216b-7517-49cc-8406-1f13632adbe9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:02:09.288916 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.288848    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9jc\" (UniqueName: \"kubernetes.io/projected/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-kube-api-access-9m9jc\") pod \"payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf\" (UID: \"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:02:09.289469 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.289452    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf\" (UID: \"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:02:09.309331 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.309283    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9jc\" (UniqueName: \"kubernetes.io/projected/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-kube-api-access-9m9jc\") pod \"payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf\" (UID: \"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:02:09.390390 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.390278    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s5wf\" (UniqueName: \"kubernetes.io/projected/b886216b-7517-49cc-8406-1f13632adbe9-kube-api-access-7s5wf\") pod \"payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct\" (UID: \"b886216b-7517-49cc-8406-1f13632adbe9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:02:09.390558 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.390406    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b886216b-7517-49cc-8406-1f13632adbe9-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct\" (UID: \"b886216b-7517-49cc-8406-1f13632adbe9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:02:09.391117 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.391097    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b886216b-7517-49cc-8406-1f13632adbe9-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct\" (UID: \"b886216b-7517-49cc-8406-1f13632adbe9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:02:09.411770 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.411744    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s5wf\" (UniqueName: \"kubernetes.io/projected/b886216b-7517-49cc-8406-1f13632adbe9-kube-api-access-7s5wf\") pod \"payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct\" (UID: \"b886216b-7517-49cc-8406-1f13632adbe9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:02:09.431637 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.431612    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:02:09.514491 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.514462    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:02:09.748769 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.748744    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"]
Nov 02 02:02:09.750919 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:02:09.750889    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5203fe3e_f50e_48a2_a0e8_c6a6ba88a728.slice/crio-6fad17de04e29da55db2f4ae10d45b1fc7acf19b486c6d92778d47446711ff77 WatchSource:0}: Error finding container 6fad17de04e29da55db2f4ae10d45b1fc7acf19b486c6d92778d47446711ff77: Status 404 returned error can't find the container with id 6fad17de04e29da55db2f4ae10d45b1fc7acf19b486c6d92778d47446711ff77
Nov 02 02:02:09.821063 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:09.821023    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"]
Nov 02 02:02:09.823852 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:02:09.823825    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb886216b_7517_49cc_8406_1f13632adbe9.slice/crio-925199ec8765302b6f30fcc59ab23166362a9bf8429ece6214b6594e3093dff6 WatchSource:0}: Error finding container 925199ec8765302b6f30fcc59ab23166362a9bf8429ece6214b6594e3093dff6: Status 404 returned error can't find the container with id 925199ec8765302b6f30fcc59ab23166362a9bf8429ece6214b6594e3093dff6
Nov 02 02:02:10.024358 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.024241    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct" event={"ID":"b886216b-7517-49cc-8406-1f13632adbe9","Type":"ContainerStarted","Data":"f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495"}
Nov 02 02:02:10.024358 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.024337    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:02:10.024624 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.024366    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct" event={"ID":"b886216b-7517-49cc-8406-1f13632adbe9","Type":"ContainerStarted","Data":"925199ec8765302b6f30fcc59ab23166362a9bf8429ece6214b6594e3093dff6"}
Nov 02 02:02:10.026090 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.026056    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6" event={"ID":"ce71f335-11a3-4c93-b7ed-efd7e001dfdc","Type":"ContainerStarted","Data":"45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04"}
Nov 02 02:02:10.026229 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.026120    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:10.026229 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.026148    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6" event={"ID":"ce71f335-11a3-4c93-b7ed-efd7e001dfdc","Type":"ContainerStarted","Data":"5049265f2e5ecc51c3b0d92808e8b886b80d23d3010040b1edfb864b4a4fa7e2"}
Nov 02 02:02:10.027669 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.027645    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf" event={"ID":"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728","Type":"ContainerStarted","Data":"6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70"}
Nov 02 02:02:10.027787 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.027684    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf" event={"ID":"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728","Type":"ContainerStarted","Data":"6fad17de04e29da55db2f4ae10d45b1fc7acf19b486c6d92778d47446711ff77"}
Nov 02 02:02:10.027787 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.027730    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:02:10.071320 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.071238    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct" podStartSLOduration=1.071223826 podStartE2EDuration="1.071223826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:10.07008399 +0000 UTC m=+2835.375017005" watchObservedRunningTime="2026-11-02 02:02:10.071223826 +0000 UTC m=+2835.376156651"
Nov 02 02:02:10.107447 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.107392    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf" podStartSLOduration=1.107377507 podStartE2EDuration="1.107377507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:10.105009855 +0000 UTC m=+2835.409942681" watchObservedRunningTime="2026-11-02 02:02:10.107377507 +0000 UTC m=+2835.412310332"
Nov 02 02:02:10.141857 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:10.141793    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6" podStartSLOduration=1.496416999 podStartE2EDuration="2.141778983s" totalImagesPullingTime="645.361984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:10.140029152 +0000 UTC m=+2835.444961978" watchObservedRunningTime="2026-11-02 02:02:10.141778983 +0000 UTC m=+2835.446711788"
Nov 02 02:02:16.057813 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:16.057777    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:02:17.683479 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:02:17.683443    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:02:21.037767 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:21.037721    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:02:21.039498 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:21.039467    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:02:21.426775 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:21.426748    4851 patch_prober.go:29] interesting pod/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": read tcp 10.132.0.2:60902->10.132.1.66:15021: read: connection reset by peer" start-of-body=
Nov 02 02:02:21.426974 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:21.426805    4851 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": read tcp 10.132.0.2:60902->10.132.1.66:15021: read: connection reset by peer"
Nov 02 02:02:22.125638 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:22.125603    4851 generic.go:356] "Generic (PLEG): container finished" podID="558da18b-bd75-4787-8619-1a4851b037bb" containerID="62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e" exitCode=0
Nov 02 02:02:22.126135 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:22.125683    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" event={"ID":"558da18b-bd75-4787-8619-1a4851b037bb","Type":"ContainerDied","Data":"62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e"}
Nov 02 02:02:22.126135 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:22.126085    4851 scope.go:122] "RemoveContainer" containerID="62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e"
Nov 02 02:02:22.127380 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:22.127352    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:02:22.127446 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:22.127425    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:02:22.127485 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:22.127458    4851 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:02:23.132127 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:23.132093    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" event={"ID":"558da18b-bd75-4787-8619-1a4851b037bb","Type":"ContainerStarted","Data":"da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f"}
Nov 02 02:02:23.555739 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:23.555704    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:23.556643 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:23.556603    4851 patch_prober.go:29] interesting pod/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 02:02:23.556785 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:23.556667    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 02:02:24.557754 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:24.557711    4851 patch_prober.go:29] interesting pod/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 02:02:24.558169 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:24.557783    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 02:02:25.557764 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:25.557723    4851 patch_prober.go:29] interesting pod/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 02:02:25.558188 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:25.557794    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 02:02:26.557760 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:26.557722    4851 patch_prober.go:29] interesting pod/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 02:02:26.557961 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:26.557783    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 02:02:27.557782 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:27.557743    4851 patch_prober.go:29] interesting pod/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 02:02:27.558004 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:27.557809    4851 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 02:02:28.559509 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:28.559478    4851 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:28.559901 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:28.559867    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:28.560465 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:02:28.560445    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:02:47.655559 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:02:47.655517    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:03:04.293545 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:04.293511    4851 scope.go:122] "RemoveContainer" containerID="6378072c15e50698cb95012af133fd6fe3b7878f11c185a947ea2f8445d35674"
Nov 02 02:03:13.034966 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:13.034924    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"]
Nov 02 02:03:13.036870 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:13.036844    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9pvjp"]
Nov 02 02:03:14.845376 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:14.845340    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e23437f-204b-4a34-9615-83ac80c0bf7b" path="/var/lib/kubelet/pods/8e23437f-204b-4a34-9615-83ac80c0bf7b/volumes"
Nov 02 02:03:17.656746 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:03:17.656710    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:03:32.028994 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:32.028903    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"]
Nov 02 02:03:32.030958 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:32.030936    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6ks4s"]
Nov 02 02:03:32.845068 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:32.845032    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="228b1282-0bfe-4e7b-a865-60da3b5d48f7" path="/var/lib/kubelet/pods/228b1282-0bfe-4e7b-a865-60da3b5d48f7/volumes"
Nov 02 02:03:47.655723 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:03:47.655618    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:03:52.878152 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:52.878120    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"]
Nov 02 02:03:52.894146 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:52.894101    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"]
Nov 02 02:03:57.879429 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:57.879378    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct" podUID="b886216b-7517-49cc-8406-1f13632adbe9" containerName="payload-processing" containerID="cri-o://f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495" gracePeriod=25
Nov 02 02:03:57.894876 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:57.894816    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf" podUID="5203fe3e-f50e-48a2-a0e8-c6a6ba88a728" containerName="payload-pre-processing" containerID="cri-o://6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70" gracePeriod=25
Nov 02 02:03:57.928869 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:57.928836    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"]
Nov 02 02:03:57.929089 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:57.929058    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6" podUID="ce71f335-11a3-4c93-b7ed-efd7e001dfdc" containerName="maas-api" containerID="cri-o://45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04" gracePeriod=30
Nov 02 02:03:58.199981 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.199952    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:03:58.250060 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.250014    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:03:58.262490 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.262464    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:03:58.264629 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.264610    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-kube-api-access-9m9jc\" (UniqueName: \"kubernetes.io/projected/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-kube-api-access-9m9jc\") pod \"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728\" (UID: \"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728\") "
Nov 02 02:03:58.264725 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.264712    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-plugins-config-volume\") pod \"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728\" (UID: \"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728\") "
Nov 02 02:03:58.265090 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.265069    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-plugins-config-volume" pod "5203fe3e-f50e-48a2-a0e8-c6a6ba88a728" (UID: "5203fe3e-f50e-48a2-a0e8-c6a6ba88a728"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:58.267231 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.267196    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-kube-api-access-9m9jc" pod "5203fe3e-f50e-48a2-a0e8-c6a6ba88a728" (UID: "5203fe3e-f50e-48a2-a0e8-c6a6ba88a728"). InnerVolumeSpecName "kube-api-access-9m9jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:58.365953 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.365926    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-kube-api-access-mph5h\" (UniqueName: \"kubernetes.io/projected/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-kube-api-access-mph5h\") pod \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") "
Nov 02 02:03:58.366136 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.366013    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b886216b-7517-49cc-8406-1f13632adbe9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b886216b-7517-49cc-8406-1f13632adbe9-plugins-config-volume\") pod \"b886216b-7517-49cc-8406-1f13632adbe9\" (UID: \"b886216b-7517-49cc-8406-1f13632adbe9\") "
Nov 02 02:03:58.366136 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.366116    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-metrics-certs\") pod \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") "
Nov 02 02:03:58.366235 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.366166    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-openshift-service-ca\") pod \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") "
Nov 02 02:03:58.366235 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.366227    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b886216b-7517-49cc-8406-1f13632adbe9-kube-api-access-7s5wf\" (UniqueName: \"kubernetes.io/projected/b886216b-7517-49cc-8406-1f13632adbe9-kube-api-access-7s5wf\") pod \"b886216b-7517-49cc-8406-1f13632adbe9\" (UID: \"b886216b-7517-49cc-8406-1f13632adbe9\") "
Nov 02 02:03:58.366401 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.366285    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-maas-api-tls\") pod \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\" (UID: \"ce71f335-11a3-4c93-b7ed-efd7e001dfdc\") "
Nov 02 02:03:58.366401 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.366349    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b886216b-7517-49cc-8406-1f13632adbe9-plugins-config-volume" pod "b886216b-7517-49cc-8406-1f13632adbe9" (UID: "b886216b-7517-49cc-8406-1f13632adbe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:58.366632 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.366597    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-openshift-service-ca" pod "ce71f335-11a3-4c93-b7ed-efd7e001dfdc" (UID: "ce71f335-11a3-4c93-b7ed-efd7e001dfdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:58.366632 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.366632    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m9jc\" (UniqueName: \"kubernetes.io/projected/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-kube-api-access-9m9jc\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:03:58.367016 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.366651    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b886216b-7517-49cc-8406-1f13632adbe9-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:03:58.367016 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.366664    4851 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728-plugins-config-volume\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:03:58.368443 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.368415    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-metrics-certs" pod "ce71f335-11a3-4c93-b7ed-efd7e001dfdc" (UID: "ce71f335-11a3-4c93-b7ed-efd7e001dfdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:58.368443 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.368434    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-kube-api-access-mph5h" pod "ce71f335-11a3-4c93-b7ed-efd7e001dfdc" (UID: "ce71f335-11a3-4c93-b7ed-efd7e001dfdc"). InnerVolumeSpecName "kube-api-access-mph5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:58.368572 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.368443    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b886216b-7517-49cc-8406-1f13632adbe9-kube-api-access-7s5wf" pod "b886216b-7517-49cc-8406-1f13632adbe9" (UID: "b886216b-7517-49cc-8406-1f13632adbe9"). InnerVolumeSpecName "kube-api-access-7s5wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:58.368662 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.368643    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-maas-api-tls" pod "ce71f335-11a3-4c93-b7ed-efd7e001dfdc" (UID: "ce71f335-11a3-4c93-b7ed-efd7e001dfdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:58.467730 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.467646    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s5wf\" (UniqueName: \"kubernetes.io/projected/b886216b-7517-49cc-8406-1f13632adbe9-kube-api-access-7s5wf\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:03:58.467730 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.467677    4851 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-maas-api-tls\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:03:58.467730 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.467687    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mph5h\" (UniqueName: \"kubernetes.io/projected/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-kube-api-access-mph5h\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:03:58.467730 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.467695    4851 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-metrics-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:03:58.467730 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.467704    4851 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce71f335-11a3-4c93-b7ed-efd7e001dfdc-openshift-service-ca\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:03:58.633463 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.633430    4851 generic.go:356] "Generic (PLEG): container finished" podID="b886216b-7517-49cc-8406-1f13632adbe9" containerID="f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495" exitCode=0
Nov 02 02:03:58.633661 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.633483    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"
Nov 02 02:03:58.633661 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.633513    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct" event={"ID":"b886216b-7517-49cc-8406-1f13632adbe9","Type":"ContainerDied","Data":"f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495"}
Nov 02 02:03:58.633661 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.633576    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct" event={"ID":"b886216b-7517-49cc-8406-1f13632adbe9","Type":"ContainerDied","Data":"925199ec8765302b6f30fcc59ab23166362a9bf8429ece6214b6594e3093dff6"}
Nov 02 02:03:58.633661 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.633608    4851 scope.go:122] "RemoveContainer" containerID="f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495"
Nov 02 02:03:58.634952 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.634926    4851 generic.go:356] "Generic (PLEG): container finished" podID="ce71f335-11a3-4c93-b7ed-efd7e001dfdc" containerID="45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04" exitCode=0
Nov 02 02:03:58.635071 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.634965    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"
Nov 02 02:03:58.635071 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.635003    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6" event={"ID":"ce71f335-11a3-4c93-b7ed-efd7e001dfdc","Type":"ContainerDied","Data":"45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04"}
Nov 02 02:03:58.635071 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.635047    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6" event={"ID":"ce71f335-11a3-4c93-b7ed-efd7e001dfdc","Type":"ContainerDied","Data":"5049265f2e5ecc51c3b0d92808e8b886b80d23d3010040b1edfb864b4a4fa7e2"}
Nov 02 02:03:58.636536 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.636517    4851 generic.go:356] "Generic (PLEG): container finished" podID="5203fe3e-f50e-48a2-a0e8-c6a6ba88a728" containerID="6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70" exitCode=0
Nov 02 02:03:58.636656 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.636563    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"
Nov 02 02:03:58.636656 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.636635    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf" event={"ID":"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728","Type":"ContainerDied","Data":"6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70"}
Nov 02 02:03:58.636777 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.636672    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf" event={"ID":"5203fe3e-f50e-48a2-a0e8-c6a6ba88a728","Type":"ContainerDied","Data":"6fad17de04e29da55db2f4ae10d45b1fc7acf19b486c6d92778d47446711ff77"}
Nov 02 02:03:58.644063 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.644042    4851 scope.go:122] "RemoveContainer" containerID="f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495"
Nov 02 02:03:58.644423 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:03:58.644394    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495\": container with ID starting with f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495 not found: ID does not exist" containerID="f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495"
Nov 02 02:03:58.644508 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.644436    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495"} err="failed to get container status \"f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495\": rpc error: code = NotFound desc = could not find container \"f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495\": container with ID starting with f6ca9efe546ea9ff6a5136e2dabc9e27feb93d976d39addc0795dedc5052b495 not found: ID does not exist"
Nov 02 02:03:58.644508 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.644458    4851 scope.go:122] "RemoveContainer" containerID="45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04"
Nov 02 02:03:58.653914 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.653897    4851 scope.go:122] "RemoveContainer" containerID="45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04"
Nov 02 02:03:58.654175 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:03:58.654156    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04\": container with ID starting with 45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04 not found: ID does not exist" containerID="45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04"
Nov 02 02:03:58.654230 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.654180    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04"} err="failed to get container status \"45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04\": rpc error: code = NotFound desc = could not find container \"45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04\": container with ID starting with 45ee401572ab512ea982ea55d3d5ed7aa2f97bd9b6d451e7d446d9fd27fd7e04 not found: ID does not exist"
Nov 02 02:03:58.654230 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.654196    4851 scope.go:122] "RemoveContainer" containerID="6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70"
Nov 02 02:03:58.662094 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.662077    4851 scope.go:122] "RemoveContainer" containerID="6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70"
Nov 02 02:03:58.662607 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:03:58.662409    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70\": container with ID starting with 6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70 not found: ID does not exist" containerID="6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70"
Nov 02 02:03:58.662607 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.662437    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70"} err="failed to get container status \"6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70\": rpc error: code = NotFound desc = could not find container \"6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70\": container with ID starting with 6dab9d9dd05489fef86c1f18da8eb3ce88361cb29aa3cefd172ff9ab83d91b70 not found: ID does not exist"
Nov 02 02:03:58.662776 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.662643    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"]
Nov 02 02:03:58.664736 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.664717    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-b12eb5-68cfc868f-r2zl6"]
Nov 02 02:03:58.679157 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.679131    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"]
Nov 02 02:03:58.685584 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.685556    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-b12eb5-794b7b9d7-vv8kf"]
Nov 02 02:03:58.696099 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.696078    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"]
Nov 02 02:03:58.701865 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.701848    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-b12eb5-85cfc4d7fc-jdlct"]
Nov 02 02:03:58.845822 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.845749    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5203fe3e-f50e-48a2-a0e8-c6a6ba88a728" path="/var/lib/kubelet/pods/5203fe3e-f50e-48a2-a0e8-c6a6ba88a728/volumes"
Nov 02 02:03:58.846122 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.846101    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b886216b-7517-49cc-8406-1f13632adbe9" path="/var/lib/kubelet/pods/b886216b-7517-49cc-8406-1f13632adbe9/volumes"
Nov 02 02:03:58.846502 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:03:58.846488    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce71f335-11a3-4c93-b7ed-efd7e001dfdc" path="/var/lib/kubelet/pods/ce71f335-11a3-4c93-b7ed-efd7e001dfdc/volumes"
Nov 02 02:04:04.997032 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:04.996994    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"]
Nov 02 02:04:04.997628 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:04.997218    4851 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy" containerID="cri-o://da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f" gracePeriod=30
Nov 02 02:04:10.313124 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.313096    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:04:10.435920 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.435883    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-socket\") pod \"558da18b-bd75-4787-8619-1a4851b037bb\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") "
Nov 02 02:04:10.436141 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.435954    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/558da18b-bd75-4787-8619-1a4851b037bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558da18b-bd75-4787-8619-1a4851b037bb-istio-podinfo\") pod \"558da18b-bd75-4787-8619-1a4851b037bb\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") "
Nov 02 02:04:10.436141 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.435982    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558da18b-bd75-4787-8619-1a4851b037bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558da18b-bd75-4787-8619-1a4851b037bb-istiod-ca-cert\") pod \"558da18b-bd75-4787-8619-1a4851b037bb\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") "
Nov 02 02:04:10.436141 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436001    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-credential-socket\") pod \"558da18b-bd75-4787-8619-1a4851b037bb\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") "
Nov 02 02:04:10.436383 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436214    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-socket" pod "558da18b-bd75-4787-8619-1a4851b037bb" (UID: "558da18b-bd75-4787-8619-1a4851b037bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:10.436383 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436238    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-envoy\") pod \"558da18b-bd75-4787-8619-1a4851b037bb\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") "
Nov 02 02:04:10.436383 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436280    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-kube-api-access-9wnj5\" (UniqueName: \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-kube-api-access-9wnj5\") pod \"558da18b-bd75-4787-8619-1a4851b037bb\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") "
Nov 02 02:04:10.436383 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436338    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-credential-socket" pod "558da18b-bd75-4787-8619-1a4851b037bb" (UID: "558da18b-bd75-4787-8619-1a4851b037bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:10.436383 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436372    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-data\") pod \"558da18b-bd75-4787-8619-1a4851b037bb\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") "
Nov 02 02:04:10.436664 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436428    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-istio-token\" (UniqueName: \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-istio-token\") pod \"558da18b-bd75-4787-8619-1a4851b037bb\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") "
Nov 02 02:04:10.436664 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436459    4851 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-certs\") pod \"558da18b-bd75-4787-8619-1a4851b037bb\" (UID: \"558da18b-bd75-4787-8619-1a4851b037bb\") "
Nov 02 02:04:10.436664 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436460    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558da18b-bd75-4787-8619-1a4851b037bb-istiod-ca-cert" pod "558da18b-bd75-4787-8619-1a4851b037bb" (UID: "558da18b-bd75-4787-8619-1a4851b037bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:10.436664 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436622    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-data" pod "558da18b-bd75-4787-8619-1a4851b037bb" (UID: "558da18b-bd75-4787-8619-1a4851b037bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:10.436905 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436802    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-certs" pod "558da18b-bd75-4787-8619-1a4851b037bb" (UID: "558da18b-bd75-4787-8619-1a4851b037bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:10.436905 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436883    4851 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-data\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:04:10.436905 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436901    4851 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-certs\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:04:10.437044 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436914    4851 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-workload-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:04:10.437044 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436927    4851 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558da18b-bd75-4787-8619-1a4851b037bb-istiod-ca-cert\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:04:10.437044 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.436944    4851 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-credential-socket\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:04:10.438558 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.438532    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/558da18b-bd75-4787-8619-1a4851b037bb-istio-podinfo" pod "558da18b-bd75-4787-8619-1a4851b037bb" (UID: "558da18b-bd75-4787-8619-1a4851b037bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:04:10.438758 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.438737    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-istio-token" pod "558da18b-bd75-4787-8619-1a4851b037bb" (UID: "558da18b-bd75-4787-8619-1a4851b037bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:10.439016 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.438999    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-envoy" pod "558da18b-bd75-4787-8619-1a4851b037bb" (UID: "558da18b-bd75-4787-8619-1a4851b037bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:10.439085 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.439069    4851 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-kube-api-access-9wnj5" pod "558da18b-bd75-4787-8619-1a4851b037bb" (UID: "558da18b-bd75-4787-8619-1a4851b037bb"). InnerVolumeSpecName "kube-api-access-9wnj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:10.537783 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.537744    4851 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558da18b-bd75-4787-8619-1a4851b037bb-istio-podinfo\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:04:10.537783 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.537778    4851 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558da18b-bd75-4787-8619-1a4851b037bb-istio-envoy\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:04:10.537783 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.537788    4851 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wnj5\" (UniqueName: \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-kube-api-access-9wnj5\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:04:10.537783 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.537797    4851 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558da18b-bd75-4787-8619-1a4851b037bb-istio-token\") on node \"ip-10-0-140-23.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 02:04:10.731389 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.731354    4851 generic.go:356] "Generic (PLEG): container finished" podID="558da18b-bd75-4787-8619-1a4851b037bb" containerID="da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f" exitCode=0
Nov 02 02:04:10.731613 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.731409    4851 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"
Nov 02 02:04:10.731613 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.731446    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" event={"ID":"558da18b-bd75-4787-8619-1a4851b037bb","Type":"ContainerDied","Data":"da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f"}
Nov 02 02:04:10.731613 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.731505    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9" event={"ID":"558da18b-bd75-4787-8619-1a4851b037bb","Type":"ContainerDied","Data":"8295d6dca158c8425b902d1d1242234e032a2767ff7b49f493d953ac9aee3415"}
Nov 02 02:04:10.731613 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.731529    4851 scope.go:122] "RemoveContainer" containerID="da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f"
Nov 02 02:04:10.749659 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.749633    4851 scope.go:122] "RemoveContainer" containerID="62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e"
Nov 02 02:04:10.766244 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.766220    4851 scope.go:122] "RemoveContainer" containerID="da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f"
Nov 02 02:04:10.766599 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:04:10.766576    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f\": container with ID starting with da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f not found: ID does not exist" containerID="da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f"
Nov 02 02:04:10.766715 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.766614    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f"} err="failed to get container status \"da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f\": rpc error: code = NotFound desc = could not find container \"da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f\": container with ID starting with da8e1f4f0402a756515625ed9c7e29df6c892140665317d3826a10d3629fee8f not found: ID does not exist"
Nov 02 02:04:10.766715 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.766640    4851 scope.go:122] "RemoveContainer" containerID="62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e"
Nov 02 02:04:10.766941 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:04:10.766924    4851 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e\": container with ID starting with 62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e not found: ID does not exist" containerID="62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e"
Nov 02 02:04:10.766997 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.766950    4851 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e"} err="failed to get container status \"62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e\": rpc error: code = NotFound desc = could not find container \"62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e\": container with ID starting with 62a3ef8a765f2e784b76597a1748a18814c13b4e44608c7a3e6fccf0c8bc657e not found: ID does not exist"
Nov 02 02:04:10.770220 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.770195    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"]
Nov 02 02:04:10.774859 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.774834    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-b12eb5-openshift-default-6bb4995ddf-g42s9"]
Nov 02 02:04:10.845207 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:10.845168    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558da18b-bd75-4787-8619-1a4851b037bb" path="/var/lib/kubelet/pods/558da18b-bd75-4787-8619-1a4851b037bb/volumes"
Nov 02 02:04:17.655847 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:04:17.655804    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:04:47.656756 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:04:47.656710    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:04:54.702587 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:54.702529    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-88cfb896-566x7_7a98bfd7-c685-408e-b963-686f01e3d7ad/authorino/0.log"
Nov 02 02:04:56.605806 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:56.605775    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 02:04:56.608526 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:04:56.608501    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 02:05:04.337766 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:04.337728    4851 scope.go:122] "RemoveContainer" containerID="97788da16bbeff60559db6f0b4e445d9e00130d6facdc18f2648dcae4c45d674"
Nov 02 02:05:04.346486 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:04.346465    4851 scope.go:122] "RemoveContainer" containerID="1893af3e23e13676e5377ad486aee7cf7b517048e84a57e58c30a6c82e1ab650"
Nov 02 02:05:06.170827 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:06.170746    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8445457bbd-cjjmh_fa4d91d9-ce2f-48f6-a164-720a5efcab51/manager/1.log"
Nov 02 02:05:08.593960 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:08.593924    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8445457bbd-cjjmh_fa4d91d9-ce2f-48f6-a164-720a5efcab51/manager/0.log"
Nov 02 02:05:09.157996 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:09.157965    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-694bbcbf67-jwgdr_2195022b-0d34-4c83-8a47-c8361b406708/maas-api/0.log"
Nov 02 02:05:09.893264 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:09.893233    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-z5zq5_5fc2408c-0e37-4e02-989a-3682209fadf6/revoke-keys/0.log"
Nov 02 02:05:10.125179 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:10.125144    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-xxgsr_91151a8c-d754-48b3-a47f-2dc07c065fb4/revoke-keys/0.log"
Nov 02 02:05:10.609583 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:10.609552    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e87e1fd5-gbrnx_5dae463f-239c-43e6-85be-0d6958a103ef/revoke-keys/0.log"
Nov 02 02:05:12.684956 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:12.684919    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-88cfb896-566x7_7a98bfd7-c685-408e-b963-686f01e3d7ad/authorino/0.log"
Nov 02 02:05:15.166787 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:15.166756    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c85779f4b-fx7dq_13bc296a-c669-4abe-8cf0-6eb24154f4a3/kube-auth-proxy/0.log"
Nov 02 02:05:15.373693 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:15.373662    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-44d86_ee1b737b-620b-4615-b94f-0ae145b61646/istio-proxy/0.log"
Nov 02 02:05:16.109342 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:16.109282    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-dvgpc_2aa0e67e-2315-4bba-8060-e7d7b101918c/istio-proxy/0.log"
Nov 02 02:05:16.489164 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:16.489130    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-kw5w7_cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c/istio-proxy/0.log"
Nov 02 02:05:16.699174 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:16.699146    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-ns4lc_0de30efc-e5bc-428c-ba8d-6b7967e29473/istio-proxy/0.log"
Nov 02 02:05:17.477620 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:17.477576    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9vj42_b5d69b63-2892-44fd-a2ba-777e1278ecc5/payload-pre-processing/0.log"
Nov 02 02:05:17.656407 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:05:17.656372    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:05:17.841362 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:17.841248    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c986fbdbf-f69k6_cfeff66f-98e2-422f-a3be-cfc9f1d7a829/router/0.log"
Nov 02 02:05:19.300778 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:19.300739    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-srgl9_d46b2d7a-fabe-4466-818c-9fc511dbd10e/main/0.log"
Nov 02 02:05:19.711814 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:19.711784    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr_d52b9b8d-07d7-49d9-b7e0-92cd937b26c6/main/0.log"
Nov 02 02:05:19.899930 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:19.899901    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-jgzjl_437397c1-b8b9-4a7a-90db-17411343e18e/main/0.log"
Nov 02 02:05:21.220644 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:21.220600    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-88cfb896-566x7_7a98bfd7-c685-408e-b963-686f01e3d7ad/authorino/0.log"
Nov 02 02:05:30.582326 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:30.582272    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8445457bbd-cjjmh_fa4d91d9-ce2f-48f6-a164-720a5efcab51/manager/1.log"
Nov 02 02:05:32.615440 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:32.615406    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-694bbcbf67-jwgdr_2195022b-0d34-4c83-8a47-c8361b406708/maas-api/0.log"
Nov 02 02:05:33.214490 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:33.214459    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-z5zq5_5fc2408c-0e37-4e02-989a-3682209fadf6/revoke-keys/0.log"
Nov 02 02:05:33.390050 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:33.390023    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-xxgsr_91151a8c-d754-48b3-a47f-2dc07c065fb4/revoke-keys/0.log"
Nov 02 02:05:33.742608 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:33.742578    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e87e1fd5-gbrnx_5dae463f-239c-43e6-85be-0d6958a103ef/revoke-keys/0.log"
Nov 02 02:05:35.300610 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:35.300575    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-88cfb896-566x7_7a98bfd7-c685-408e-b963-686f01e3d7ad/authorino/0.log"
Nov 02 02:05:36.023316 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:36.023271    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"]
Nov 02 02:05:36.025910 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:36.025884    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xxgsr"]
Nov 02 02:05:36.844547 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:36.844508    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91151a8c-d754-48b3-a47f-2dc07c065fb4" path="/var/lib/kubelet/pods/91151a8c-d754-48b3-a47f-2dc07c065fb4/volumes"
Nov 02 02:05:37.617557 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:37.617527    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c85779f4b-fx7dq_13bc296a-c669-4abe-8cf0-6eb24154f4a3/kube-auth-proxy/0.log"
Nov 02 02:05:37.793950 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:37.793902    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-44d86_ee1b737b-620b-4615-b94f-0ae145b61646/istio-proxy/0.log"
Nov 02 02:05:38.549835 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:38.549799    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-dvgpc_2aa0e67e-2315-4bba-8060-e7d7b101918c/istio-proxy/0.log"
Nov 02 02:05:38.915503 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:38.915474    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-kw5w7_cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c/istio-proxy/0.log"
Nov 02 02:05:39.101821 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:39.101784    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-ns4lc_0de30efc-e5bc-428c-ba8d-6b7967e29473/istio-proxy/0.log"
Nov 02 02:05:39.857103 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:39.857066    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9vj42_b5d69b63-2892-44fd-a2ba-777e1278ecc5/payload-pre-processing/0.log"
Nov 02 02:05:40.211742 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:40.211716    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c986fbdbf-f69k6_cfeff66f-98e2-422f-a3be-cfc9f1d7a829/router/0.log"
Nov 02 02:05:41.074849 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:41.074817    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-srgl9_d46b2d7a-fabe-4466-818c-9fc511dbd10e/main/0.log"
Nov 02 02:05:41.425840 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:41.425799    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xkmzr_d52b9b8d-07d7-49d9-b7e0-92cd937b26c6/main/0.log"
Nov 02 02:05:41.614350 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:41.614319    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-jgzjl_437397c1-b8b9-4a7a-90db-17411343e18e/main/0.log"
Nov 02 02:05:47.655551 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:05:47.655513    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:05:53.021178 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:53.021145    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"]
Nov 02 02:05:53.023949 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:53.023923    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-z5zq5"]
Nov 02 02:05:54.845273 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:05:54.845234    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc2408c-0e37-4e02-989a-3682209fadf6" path="/var/lib/kubelet/pods/5fc2408c-0e37-4e02-989a-3682209fadf6/volumes"
Nov 02 02:06:12.028355 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:06:12.028276    4851 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"]
Nov 02 02:06:12.031252 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:06:12.031227    4851 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e87e1fd5-gbrnx"]
Nov 02 02:06:12.844798 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:06:12.844758    4851 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dae463f-239c-43e6-85be-0d6958a103ef" path="/var/lib/kubelet/pods/5dae463f-239c-43e6-85be-0d6958a103ef/volumes"
Nov 02 02:06:17.655990 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:06:17.655954    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:06:47.655752 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:06:47.655664    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:07:04.379405 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:04.379372    4851 scope.go:122] "RemoveContainer" containerID="10037cea532b32005b0628614146d61d7ffa9de16c04b981fffec6e02776c319"
Nov 02 02:07:04.391690 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:04.391668    4851 scope.go:122] "RemoveContainer" containerID="b1ab088c71f99366d75e28a3ee4459fdeb8e5c27f40bf922af039a3b40760486"
Nov 02 02:07:08.878450 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878414    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zr67k/must-gather-kxpnz"]
Nov 02 02:07:08.878864 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878801    4851 cpu_manager.go:414] "containerMap: removing container" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy"
Nov 02 02:07:08.878864 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878812    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy"
Nov 02 02:07:08.878864 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878826    4851 cpu_manager.go:414] "containerMap: removing container" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy"
Nov 02 02:07:08.878864 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878831    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy"
Nov 02 02:07:08.878864 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878841    4851 cpu_manager.go:414] "containerMap: removing container" podUID="b886216b-7517-49cc-8406-1f13632adbe9" containerName="payload-processing"
Nov 02 02:07:08.878864 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878845    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b886216b-7517-49cc-8406-1f13632adbe9" containerName="payload-processing"
Nov 02 02:07:08.878864 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878861    4851 cpu_manager.go:414] "containerMap: removing container" podUID="ce71f335-11a3-4c93-b7ed-efd7e001dfdc" containerName="maas-api"
Nov 02 02:07:08.878864 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878865    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce71f335-11a3-4c93-b7ed-efd7e001dfdc" containerName="maas-api"
Nov 02 02:07:08.879139 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878877    4851 cpu_manager.go:414] "containerMap: removing container" podUID="5203fe3e-f50e-48a2-a0e8-c6a6ba88a728" containerName="payload-pre-processing"
Nov 02 02:07:08.879139 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878882    4851 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5203fe3e-f50e-48a2-a0e8-c6a6ba88a728" containerName="payload-pre-processing"
Nov 02 02:07:08.879139 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878949    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce71f335-11a3-4c93-b7ed-efd7e001dfdc" containerName="maas-api"
Nov 02 02:07:08.879139 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878957    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="5203fe3e-f50e-48a2-a0e8-c6a6ba88a728" containerName="payload-pre-processing"
Nov 02 02:07:08.879139 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878967    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy"
Nov 02 02:07:08.879139 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878972    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="558da18b-bd75-4787-8619-1a4851b037bb" containerName="istio-proxy"
Nov 02 02:07:08.879139 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.878979    4851 memory_manager.go:365] "RemoveStaleState removing state" podUID="b886216b-7517-49cc-8406-1f13632adbe9" containerName="payload-processing"
Nov 02 02:07:08.883635 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.883615    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zr67k/must-gather-kxpnz"
Nov 02 02:07:08.887566 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.887541    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zr67k\"/\"openshift-service-ca.crt\""
Nov 02 02:07:08.888762 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.888739    4851 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zr67k\"/\"kube-root-ca.crt\""
Nov 02 02:07:08.888762 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.888739    4851 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zr67k\"/\"default-dockercfg-89bfr\""
Nov 02 02:07:08.900709 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.900684    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zr67k/must-gather-kxpnz"]
Nov 02 02:07:08.992814 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.992775    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4px\" (UniqueName: \"kubernetes.io/projected/667d23de-649d-41f4-be36-152d91203b3f-kube-api-access-7b4px\") pod \"must-gather-kxpnz\" (UID: \"667d23de-649d-41f4-be36-152d91203b3f\") " pod="openshift-must-gather-zr67k/must-gather-kxpnz"
Nov 02 02:07:08.993093 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:08.993069    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/667d23de-649d-41f4-be36-152d91203b3f-must-gather-output\") pod \"must-gather-kxpnz\" (UID: \"667d23de-649d-41f4-be36-152d91203b3f\") " pod="openshift-must-gather-zr67k/must-gather-kxpnz"
Nov 02 02:07:09.094795 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:09.094752    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4px\" (UniqueName: \"kubernetes.io/projected/667d23de-649d-41f4-be36-152d91203b3f-kube-api-access-7b4px\") pod \"must-gather-kxpnz\" (UID: \"667d23de-649d-41f4-be36-152d91203b3f\") " pod="openshift-must-gather-zr67k/must-gather-kxpnz"
Nov 02 02:07:09.094795 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:09.094802    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/667d23de-649d-41f4-be36-152d91203b3f-must-gather-output\") pod \"must-gather-kxpnz\" (UID: \"667d23de-649d-41f4-be36-152d91203b3f\") " pod="openshift-must-gather-zr67k/must-gather-kxpnz"
Nov 02 02:07:09.095113 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:09.095096    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/667d23de-649d-41f4-be36-152d91203b3f-must-gather-output\") pod \"must-gather-kxpnz\" (UID: \"667d23de-649d-41f4-be36-152d91203b3f\") " pod="openshift-must-gather-zr67k/must-gather-kxpnz"
Nov 02 02:07:09.107026 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:09.106991    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4px\" (UniqueName: \"kubernetes.io/projected/667d23de-649d-41f4-be36-152d91203b3f-kube-api-access-7b4px\") pod \"must-gather-kxpnz\" (UID: \"667d23de-649d-41f4-be36-152d91203b3f\") " pod="openshift-must-gather-zr67k/must-gather-kxpnz"
Nov 02 02:07:09.198055 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:09.198022    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zr67k/must-gather-kxpnz"
Nov 02 02:07:09.474317 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:09.474275    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zr67k/must-gather-kxpnz"]
Nov 02 02:07:09.476971 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:07:09.476932    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod667d23de_649d_41f4_be36_152d91203b3f.slice/crio-655ffe1889ba1d6d31cb44581c87110a02012023c09e4070910466eb8f48594a WatchSource:0}: Error finding container 655ffe1889ba1d6d31cb44581c87110a02012023c09e4070910466eb8f48594a: Status 404 returned error can't find the container with id 655ffe1889ba1d6d31cb44581c87110a02012023c09e4070910466eb8f48594a
Nov 02 02:07:09.478960 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:09.478938    4851 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:07:09.794393 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:09.794312    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zr67k/must-gather-kxpnz" event={"ID":"667d23de-649d-41f4-be36-152d91203b3f","Type":"ContainerStarted","Data":"655ffe1889ba1d6d31cb44581c87110a02012023c09e4070910466eb8f48594a"}
Nov 02 02:07:11.807529 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:11.807487    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zr67k/must-gather-kxpnz" event={"ID":"667d23de-649d-41f4-be36-152d91203b3f","Type":"ContainerStarted","Data":"bb96f26c9fcd73a5d7e859eb2a35d6501647022e31018cf5d1b4ce7e5ddf8481"}
Nov 02 02:07:11.808098 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:11.807543    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zr67k/must-gather-kxpnz" event={"ID":"667d23de-649d-41f4-be36-152d91203b3f","Type":"ContainerStarted","Data":"e6d094d3f4e458a43794409aed113c1d3b59717780ab20ff42bd861887ee3cc9"}
Nov 02 02:07:11.825599 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:11.825525    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zr67k/must-gather-kxpnz" podStartSLOduration=2.826278834 podStartE2EDuration="3.825506071s" totalImagesPullingTime="999.227237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:11.824180053 +0000 UTC m=+3137.129112880" watchObservedRunningTime="2026-11-02 02:07:11.825506071 +0000 UTC m=+3137.130438898"
Nov 02 02:07:12.908214 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:12.908177    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nm87s_7b6da542-d39f-4a29-9849-08164ec387e0/global-pull-secret-syncer/0.log"
Nov 02 02:07:13.021193 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:13.021165    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5jpp9_50991da9-4137-467f-a695-0d0d4203321f/connection-checker/0.log"
Nov 02 02:07:13.052163 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:13.052126    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jhbp5_8d515403-65e0-4198-b78a-794ef628bc8f/connection-checker/0.log"
Nov 02 02:07:13.083737 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:13.083710    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-p9c2l_975d3545-32e4-4d97-a759-d4251bb28224/connection-checker/0.log"
Nov 02 02:07:13.140779 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:13.140743    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dqdrz_96b08fc5-cc99-4228-9a22-1e57c7fed18b/konnectivity-agent/0.log"
Nov 02 02:07:13.334322 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:13.334258    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-23.us-east-2.compute.internal_a10306333dfa6bc03e77a9c914cffe89/haproxy/0.log"
Nov 02 02:07:17.708039 ip-10-0-140-23 kubenswrapper[4851]: E1102 02:07:17.708003    4851 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m4wnq" containerName="extractor"
Nov 02 02:07:20.135949 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:20.135252    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-88cfb896-566x7_7a98bfd7-c685-408e-b963-686f01e3d7ad/authorino/0.log"
Nov 02 02:07:21.987876 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:21.987846    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-gsw4b_e57114c7-9426-4d83-870a-9d033cf335e4/cluster-monitoring-operator/0.log"
Nov 02 02:07:22.351892 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:22.351816    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wz5wp_deecdad1-584c-40c7-827b-17738a003035/node-exporter/0.log"
Nov 02 02:07:22.375147 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:22.375110    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wz5wp_deecdad1-584c-40c7-827b-17738a003035/kube-rbac-proxy/0.log"
Nov 02 02:07:22.399577 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:22.399552    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wz5wp_deecdad1-584c-40c7-827b-17738a003035/init-textfile/0.log"
Nov 02 02:07:22.994184 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:22.994148    4851 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"]
Nov 02 02:07:23.001687 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.001647    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.006333 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.005829    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"]
Nov 02 02:07:23.072070 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.072032    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-podres\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.072070 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.072076    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-lib-modules\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.072398 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.072114    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-proc\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.072398 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.072211    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx2pv\" (UniqueName: \"kubernetes.io/projected/faf1d8a8-791e-4300-8002-0aef5946a207-kube-api-access-nx2pv\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.072398 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.072243    4851 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-sys\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.173617 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.173048    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-podres\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.173617 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.173111    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-lib-modules\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.173617 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.173154    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-proc\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.173617 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.173196    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx2pv\" (UniqueName: \"kubernetes.io/projected/faf1d8a8-791e-4300-8002-0aef5946a207-kube-api-access-nx2pv\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.173617 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.173221    4851 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-sys\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.173617 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.173402    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-sys\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.173617 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.173485    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-podres\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.173617 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.173545    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-lib-modules\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.173617 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.173589    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faf1d8a8-791e-4300-8002-0aef5946a207-proc\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.186933 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.186901    4851 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx2pv\" (UniqueName: \"kubernetes.io/projected/faf1d8a8-791e-4300-8002-0aef5946a207-kube-api-access-nx2pv\") pod \"perf-node-gather-daemonset-km57b\" (UID: \"faf1d8a8-791e-4300-8002-0aef5946a207\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.331821 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.331734    4851 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.626496 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.626466    4851 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"]
Nov 02 02:07:23.628481 ip-10-0-140-23 kubenswrapper[4851]: W1102 02:07:23.628448    4851 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfaf1d8a8_791e_4300_8002_0aef5946a207.slice/crio-0051f6aea75ca6525d2d44b93dbeee2320e3f2d7f4b2e6cbd3a6aaafce2fc481 WatchSource:0}: Error finding container 0051f6aea75ca6525d2d44b93dbeee2320e3f2d7f4b2e6cbd3a6aaafce2fc481: Status 404 returned error can't find the container with id 0051f6aea75ca6525d2d44b93dbeee2320e3f2d7f4b2e6cbd3a6aaafce2fc481
Nov 02 02:07:23.952950 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.952913    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b" event={"ID":"faf1d8a8-791e-4300-8002-0aef5946a207","Type":"ContainerStarted","Data":"de0767b891b97e26c6267f7675f51720ff95693036044c71ac26773eaa2f623b"}
Nov 02 02:07:23.953189 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.952980    4851 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b" event={"ID":"faf1d8a8-791e-4300-8002-0aef5946a207","Type":"ContainerStarted","Data":"0051f6aea75ca6525d2d44b93dbeee2320e3f2d7f4b2e6cbd3a6aaafce2fc481"}
Nov 02 02:07:23.953189 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.953021    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:23.973220 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:23.973159    4851 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b" podStartSLOduration=1.973128351 podStartE2EDuration="1.973128351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:23.97195951 +0000 UTC m=+3149.276892340" watchObservedRunningTime="2026-11-02 02:07:23.973128351 +0000 UTC m=+3149.278061178"
Nov 02 02:07:24.211204 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:24.211118    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-9fbg8_b2edcda9-40c1-4565-8843-1083f0f60b5a/networking-console-plugin/0.log"
Nov 02 02:07:24.832929 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:24.832901    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/2.log"
Nov 02 02:07:24.838236 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:24.838213    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cqpmz_b395656d-d1a0-4cea-9b61-41788ab06732/console-operator/3.log"
Nov 02 02:07:25.940046 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:25.940014    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-b4llv_4b7adf12-c300-4019-8425-6115d6ebca69/volume-data-source-validator/0.log"
Nov 02 02:07:27.090806 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:27.090772    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7zmf4_1be28aae-f92c-4eb1-b063-f84b46967f3b/dns/0.log"
Nov 02 02:07:27.115043 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:27.115013    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7zmf4_1be28aae-f92c-4eb1-b063-f84b46967f3b/kube-rbac-proxy/0.log"
Nov 02 02:07:27.250034 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:27.250005    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2wxd5_b9c0fff4-df96-4db9-a0b1-ddcb4c8d028f/dns-node-resolver/0.log"
Nov 02 02:07:28.035243 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:28.035210    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlw47_c4afb19d-ffe5-4011-81a0-5761db34bd75/node-ca/0.log"
Nov 02 02:07:29.970694 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:29.970666    4851 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-km57b"
Nov 02 02:07:31.363381 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:31.363337    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c85779f4b-fx7dq_13bc296a-c669-4abe-8cf0-6eb24154f4a3/kube-auth-proxy/0.log"
Nov 02 02:07:31.399450 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:31.399412    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-44d86_ee1b737b-620b-4615-b94f-0ae145b61646/istio-proxy/0.log"
Nov 02 02:07:31.521136 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:31.521109    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-dvgpc_2aa0e67e-2315-4bba-8060-e7d7b101918c/istio-proxy/0.log"
Nov 02 02:07:31.586416 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:31.586387    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-kw5w7_cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c/istio-proxy/0.log"
Nov 02 02:07:31.619361 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:31.616381    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-ns4lc_0de30efc-e5bc-428c-ba8d-6b7967e29473/istio-proxy/0.log"
Nov 02 02:07:31.751882 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:31.751840    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9vj42_b5d69b63-2892-44fd-a2ba-777e1278ecc5/payload-pre-processing/0.log"
Nov 02 02:07:31.837666 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:31.837638    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c986fbdbf-f69k6_cfeff66f-98e2-422f-a3be-cfc9f1d7a829/router/0.log"
Nov 02 02:07:34.329564 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:34.329533    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wlvr8_51025d9a-739c-4934-9f5b-5bcbeb0d81ca/serve-healthcheck-canary/0.log"
Nov 02 02:07:34.906020 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:34.905993    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-bdwzr_16b92078-b559-44f6-85b0-4f9d1da9ddcf/insights-operator/0.log"
Nov 02 02:07:34.907100 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:34.907059    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-bdwzr_16b92078-b559-44f6-85b0-4f9d1da9ddcf/insights-operator/1.log"
Nov 02 02:07:35.105672 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:35.105646    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4wnq_ab7363cd-3783-4f09-b5de-ce703edde1d5/kube-rbac-proxy/0.log"
Nov 02 02:07:35.130348 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:35.130320    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4wnq_ab7363cd-3783-4f09-b5de-ce703edde1d5/exporter/0.log"
Nov 02 02:07:35.156652 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:35.156559    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4wnq_ab7363cd-3783-4f09-b5de-ce703edde1d5/extractor/0.log"
Nov 02 02:07:35.195633 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:35.195603    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-dlztf-6fpck_52de128e-db62-4aca-87e6-2af6c1c90a26/insights-gathering/0.log"
Nov 02 02:07:36.048551 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:36.048524    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c85779f4b-fx7dq_13bc296a-c669-4abe-8cf0-6eb24154f4a3/kube-auth-proxy/0.log"
Nov 02 02:07:36.068351 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:36.068326    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-44d86_ee1b737b-620b-4615-b94f-0ae145b61646/istio-proxy/0.log"
Nov 02 02:07:36.141468 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:36.141438    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-dvgpc_2aa0e67e-2315-4bba-8060-e7d7b101918c/istio-proxy/0.log"
Nov 02 02:07:36.186003 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:36.185956    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-kw5w7_cbdfb1d0-b327-48d5-8b80-f8ad7a991a1c/istio-proxy/0.log"
Nov 02 02:07:36.203392 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:36.203363    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-ns4lc_0de30efc-e5bc-428c-ba8d-6b7967e29473/istio-proxy/0.log"
Nov 02 02:07:36.270940 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:36.270915    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9vj42_b5d69b63-2892-44fd-a2ba-777e1278ecc5/payload-pre-processing/0.log"
Nov 02 02:07:36.307031 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:36.306955    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c986fbdbf-f69k6_cfeff66f-98e2-422f-a3be-cfc9f1d7a829/router/0.log"
Nov 02 02:07:37.589152 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:37.589119    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d6cbb5df9-phmfn_592ace31-a3ce-42ef-b168-64712a32a593/manager/0.log"
Nov 02 02:07:37.620740 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:37.620714    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d6cbb5df9-phmfn_592ace31-a3ce-42ef-b168-64712a32a593/copy-manifests/0.log"
Nov 02 02:07:38.952801 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:38.952774    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-5b9f7dd4f5-84bht_8286a9d3-cdfb-4e96-b7cf-e61f6f5c2f8b/manager/0.log"
Nov 02 02:07:39.629559 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:39.629529    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8445457bbd-cjjmh_fa4d91d9-ce2f-48f6-a164-720a5efcab51/manager/1.log"
Nov 02 02:07:39.634815 ip-10-0-140-23 kubenswrapper[4851]: I1102 02:07:39.634772    4851 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8445457bbd-cjjmh_fa4d91d9-ce2f-48f6-a164-720a5efcab51/manager/0.log"